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ABSTRACT 

This workbook is designed for administrators with 
project responsibilities that are to be evaluated. The purpose is to 
help the administrator understand the factors that can affect 
evaluation use and organize for the evaluation to maximize the 
potential use of the information the evaluation produces. Four phases 
of organizing an evaluation to maximize its potential for use are 
discussed: (1) delineating the context factors that provide the 
evaluation framework, (2) identifying and organizing the 
participants, (3) operationalizing the interactive process, and (4) 
communicating the findings to users. Sample worksheets are included 
for the four phases. (PN) 
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I. INTRODUCTION 



CHAPTER I 
INTRODUCTION 

Congratulations! We understand that a project for which you have 
administrative responsibility Is going to be evaluated. We know that 
you will want to make the most of this opportunity to see that the 
program functions as well as possible. Perhaps you've already worked 
with an e valuator but want to make sure that this time you get some 
mileage from the evaluation findings, this time the evaluation amounts 
to something more than a report gathering dust on the shelf, this time 
the evaluation Is used for practical purposes. Or perhaps this Is your 
first experience with evaluation? 

Either way, this workbook Is designed for you. It will acquaint 
(or reacqualnt) you with the nature of evaluation and focus your efforts 
so that you can actively direct and participate In the unfolding process. 
You will not learn here specific techniques for performing the evaluation 
- - but, of course, you don't need to know those details. Instead, our 
purpose Is to help you organize for the evaluation so you can maximize 
the potential use of the Information It produces. We believe that 
well -conceived and well -organized evaluations can lead to purposeful, 
planned action and that you, as 'administrator, are In the best position 
to make sure that evaluation use happens. 
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You Can Make A Difference ! 

Time and time again, research has shown that it is the presence of 
a person in a position of authority who takes a personal interest in the 
evaluation that makes the difference in whether or not results of the 
evaluation are actually put to use. In our view, you are that person. 

"But," you may say, "I am only one person. How can I make a 
difference? After all, the project being evaluated is only one of my 
many responsibilties." Don't worry, we're not implying that you have to 
do everything yourself! This workbook will show you how to structure 
the evaluation process so that other people can be involved to best 
advantage. Increasing the participation of the people most directly 
concerned with a project's evaluation increases the likelihood that the 
information produced by the evaluation will ultimately be used. 

Let's stop for a moment and consider who is meant by "the people 
most directly concerned with a project's evaluation." When you look at 
the overall structure of your project, who are the people who make it 
work? Perhaps there is a designated project director. Perhaps there 
are senior or other staff members with specific project responsibilities. 
Perhaps there is even a community advisory board or some other citizens' 
group with an active stake or interest in the project's services. All 
of these people have a professional and/or personal interest in the 
information produced by the evaluation - - and all of them are potential 
users of that information. 

What Is Evaluation Use? 

Since we've referred several times to the concept of "evaluation 
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use/ a clear definition of the term is called for. By "evaluation 
use" wa mean the purposeful, planned action that results from applying 
evaluation information to the problem, question, or concern at hand. 
Let's illustrate this concept with some examples. 

Imagine that you are the director of a districtwide bilingual education 
project. Your project takes a native-language-first approach to the teaching 
of reading; that is, children in elementary grades are taught to read in 
their native language (Spanish) before they are taught to read in English. 
You have some question about the effectiveness of this approach, so you 
ask the evaluator to gather specific data on this evaluation question. 
When the data becomes available, you call a meeting of project staff to 
discuss the findings. At this meeting, you and the staff reach a decision 
about whether to continue with the native-language-first approach or to 
change to an English-language-first approach. In this way, the evaluation 
has been "used"; that is, a purposeful action has resulted from the 
application of evaluation information to the question of concern. 

Let's take another example. Suppose that you are a corporation vice- 
president in charge of sales operations. One of your regional divisions 
has maintained a consistently outstanding sales record. You want to know 
what strategies contribute to the success of this division, and whether 
you should institute training workshops to introduce these successful 
strategies to the sales personnel of other regions. So you ask the 
evaluator to look at the regional division with an eye to identifying the ' 
characteristics that make it successful. After studying the data on this 
point that emerges from the evaluation, you conclude that the region's 
success is attributable to a unique combination of personalities and 
contexts that is not transferable to other regions. So you decide not to 
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Institute the proposed training sessions. Once again, evaluation has 

been "used 11 ; that is, a purposeful action has resulted from the application 

of evaluation information to the question of concern. 

Notice that both of the uses of evaluation just described can be 
called "action uses". "Action" is one type of possible evaluation use. 
It 1s the most obvious type of evaluation use to identify since it usually 
involves a specific decision or project modification. But there can also 
be another type of evaluation use. 

The second major type of evaluation use is often called "conceptual 
use". To illustrate it, let's return to our examples. Once again, put 
yourself in the place of the bilingual project administrator. You've been 
under fire from some of your English-language teachers as well as from 
some community members who don't believe that the native-language-first 
approach is appropriate. So you ask the evaluator to collect information 
on whether the native-language-first approach hinders or promotes student 
progress. Then you present this information at one of the meetings of the 
community advisory council so that the members can, in turn, communicate 
the dialogue to the larger parent community. 

In this instance, neither a specific decision nor a clear-cut action 
resulted from the use of the evaluation information. What did result, 
however, was an attempt to increase understanding, change attitude or 
modify beliefs. This is close to a definition of "conceptual use": 
the intentional influencing or modifying of an individual's or a group's 
understanding of, attitudes toward, or beliefs about something. 

Let's look again at the corporation example. Remember that you are 
the vice-president for^sales operations. Suppose that morale is low in 
q some sales regions because the successful region so consistently outperforms 
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them. The sales personnel in these regions have poor attitudes and 
diminished self-confidence. So you ask the evaluator to provide 
information on what makes the first region so successful. The evaluation 
indicates that the factors responsible for success are idiosyncratic 
to the region and would be impossible to duplicate. You pass this 
information on to the sales staff, emphasizing that comparisons between 
the successful region and other regions are pointless. You intend that 
the information will build the self-confidence of demoralized staff 
members so that they will renew their sales efforts on the basis of 
increased belief in their own potential effectiveness. In this instance, 
then, what resulted from the application of evaluation information was 
not a specific action but rather an increase in understanding and a 
consequent change (for the better, one hopes) in attitudes and beliefs. 
Any such change should ultimately result in increases in the sales efforts 
of the personnel from the other regions. 

As you can see, identifying instances of conceptual use is more 
difficult than identifying instances of action use. To make it easier, you 
should look at the personal interactions and information exchanges that 
take place during the course of an evaluation, both between yourself and 
the evaluator, and between the evaluator and the other people who are 
potential users of evaluation information. When these interactions and 
exchanges result in attitude changes, you are probably seeing conceptual 
use of the evaluation. 

RECAP : Let's recap our discussion to this point. So far, we've said 
that you, in concert with your evaluator, are the person most likely 
to succeed in making evaluation use happen. We've identified other 
potential users of evaluation information: namely, program staff and 
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concerned individuals outside the organization (e.g., board 
members). And finally, we've defined two major types of evaluation 
use - action and conceptual - toward which you can be aiming as you 
consider the most important topic of all - - what you can do to make 
use happen. 

What Can You Do? 

Early on in the discussion, we said that we'd indicate how you can 
structure the evaluation process to best advantage. Well, we've now come 
to that point. There are two imperatives here: you must (1) understand 
the factors that can affect evaluation use; and (2) organize to maximize 
the (positive) effect of these factors. Let's take each point in turn. 

• Understand important factors 

The key to understanding your role in structuring the evaluation 
process is understanding the multiplicity of factors which can affect 
evaluation use. What do we mean by "factor"? In this context, a factor is 
any characteristic or element present in a given situation which can affect 
the outcome of that situation. 

Let's illustrate, by looking at one set of factors. You, the admin- 
istrator, are a living human being. You come to the evaluation equipped 
with certain attitudes, beliefs, and other personal characteristics, each 
of which can affect your dealings with the evaluator and with other 
potential users, as well as the use you make of evaluation information. 
All the other potential users come equipped with their own characteristics 
and beliefs, which in turn can : affect their dealings with you and the 
evaluator and their use of evaluation information. 
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For the sake of simplicity, all factors can be grouped Into three 
major categories: human factors, context factors, and evaluation factors ♦ 
To simplify further your task of dealing with these factors, this workbook 
focuses only on these factors which research shows to be most closely 
related to the use of evaluation Information. 

By human factors , we mean the characteristics and beliefs of the people 
associated with the project being evaluated, Including the evaluator. The 
term also refers to the manner of the interaction between evaluator and 
potential users. Some examples of these factors are people's attitudes 
toward the evaluator and toward the project* the professional backgrounds of 
project staff, and the evaluator 1 s rapport with potential users. 

The term context factors refers to elements or components of the 
situational backdrop against which the evaluation process unfolds. Examples 
of context factors are fiscal or other constraints on the evaluation, the 
social and political climate surrounding the project being evaluated, and 
the degree of autonomy which the project enjoys. 

Finally, the term evaluation factors refers to specific elements of 
the evaluation procedure itself. Included here are such factors as the 
kinds of evaluative information collected, the relevance of the information 
to potential users, and the quality of the interactions between the evaluator 
and other Interested parties. 

• Organize to maximize factor effect 

As mentioned earlier, all three categories of factors can affect 
evaluation use. Now let's talk for a moment about your relationship to 
these factors. As an administrator, you can have a certain amount of 
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Influence over how these factors affect the process and outcome of 
evaluation, though just how much varies, depending on the factor and 
the specific situation. 

To illustrate, let's consider the factor that we've termed "people's 
attitudes toward the evaluator". A negative attitude will probably mean 
less chance of evaluation use; a positive attitude, more. What can you 
do to influence how people perceive the evaluator? You can probably come 
up with several suggestions: Perhaps you're thinking that you can 
formally introduce the evaluator to the potential users before the evaluation 
actually starts; or you can accompany the evaluator on his/her initial 
site visit and make informal introductions at that time; or you can give 
all potential users a written resume of the evaluator's previous accomplish- 
ments, at the same time indicating your approval of the evaluator's 
qualifications for the particular job he/she will be performing for your 
project. These are just three ways in which you can have influence 
over this factor, and we've barely scratched the surface of possible 
strategies you could use. 

Of course, certain factors are more amenable to your attempts at 
influence than others. For example, if the outside agency funding your 
project imposes certain evaluation requirements, you probably can't do 
much to change them. But even tbough your influence on some factors may be 
limited, it is important that you consider carefully your relationship 
to each factor, to make sure that you have exhausted all reasonable 
possibilities of influence before moving on. 

Speaking of moving on, we think you are now ready to get down to some 
specifics about your own project evaluation. Remember, the discussion will 
emphasize what you can do to increase the likelihood that the evaluation 
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Information 1s used. Throughout the discussion , you should keep firmly 
1n mind the purposes of your project, staff responsibilities, participant 
activities, and the range of potential users of the evaluation Information 
to be produced. 

You will find a set of sample worksheets at the beginning of each 
of the four remaining chapters. We suggest that you proceed directly 
to the text of each chapter rather than stopping to read through the 
worksheets first. The text will take you back, step by step, through 
each of the worksheets 1n due course. 

Before you turn to Chapter II, let us give you a brief overview of 
how the rest of this workbook 1s organized. As we see 1t, the task of 
the administrator ~ 1n organizing an evaluation to maximize Its potential 
for use — falls Into four phases: setting the stage (Chapter II); 
Identifying and organizing the participants (Chapter III); operational 1z1ng 
the Interactive process (Chapter IV); and adding the finishing touches 
(Chapter V). Moreover, each of these phases 1s associated with a particular 
factor cluster; that 1s, a set of factors that must be attended to 1f 
the evajuatlon 1s to realize Its potential for use. The four phases, and 
the factor clusters associated with each, are presented d1agramat1cally 
1n Figure 1. 
(Insert Figure 1 Here) 

We will be considering each of these factor clusters 1n turn. 
Please turn to Chapter II and let's get started. 
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II. SETTING THE STAGE: FACTOR CLUSTER A 
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Factor Pattern For Evaluation Use 

A. Setting the Stage 

Pre-existing evaluation bounds 

User identity * 

Program characteristics 

Irttra-organizational features 
External features 

B. Identifying/Organizing the Participants 

User interest in evaluation 

User commitment to use 

Evaluator characteristics 

• background/identity 
•commitment to use 

. • willingness to involve user in evaluation 

• choice of role 

• political sensitivity 

• credibility 

Evaluation procedures— plan 

User professional style(s) 

C. Operationalizing the Interactive Process 

Evaluation procedures—execution 

Substance of evaluation information 

Evaluator commitment to use 

Information dialogue— formative 

User information processing preferences 

D. Adding the Finishing Touches 

Evaluation reporting 

Evaluator characteristics (selected) 
Information dialogue— summative 
User comrfritment to use 
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Organizing For Evaluation Use 

Worksheet A1 



Evaluation Topic: 



Factor Cluster A: 



Pre-existing evaluation bounds 



User identity 



Program characteristics 



Intra-organizational features 



External features 



With respect to my own program and the above topic, I nee d to keep the following in mind: 
Factor Affecting Use: Relevant Information: Things I Can Do To Influence Use: 



A-1. Pre-existing 
•valuation bounds 

(written requirements; 
other contractual obliga- 
tions; fiscal constraints) 



1. 



3. 




Organizing For Evaluation Use 




Evaluation Topic: 



Factor Cluster A: 



Pre-existing evaluation bounds 



User identity 



Program characteristics 



Intra-organizational features 



External features 



With respect to my own program and the above topic, I need to keep the following in mind: 



Factor Affecting Use: Relevant Information: 



A-2. User identity 

(range; organizational 
positions; professional 
experience levels) 



Things I Can Do To Influence Jae: 



3. 



3. 



4. 
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Organizing For Evaluation Use 

Worksheet A3 



Evaluation Topic: 



Factor Cluster A: 



Pre-existing evaluation bounds 



User identity 



Program characterises 



Intra-organizational features 



External features 



With respect to my own program and the above topic, I need to keep the following in mind: 



Factor Affecting Use: Relevant Information: 



A*3. Program 
eharacteriatlca 
(age; maturity; innova- 
ttveness; overlap with 
xtther programs) 



Things I Can Do To Influence Use: 



1. 



Organizing For Evaluation Use 




Evaluation Topic: 



Factor Cluster A: 



Preexisting evaluation bounds 



User identity 



Program characteristics 



tntra-organlzatlonal features 



External features 



With respect to my own program and the above topic, I need to keep the following in mind: 



Factor Affecting Use: Relevant Information: 



A«4. mtraorgantxatlonal 
features 

(central/district office 
roles & relationships; 
institutional arrangements 
& autonomy; other likely 
kinds of information; 
perceived institutional 
& financial risk) 



1. 



Things I Can Do To Influence Use: 



1. 



2. 



3. 



4. 
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Organizing For Evaluation Use 




Evaluation Topic: 



Factor Cluster A: 



Pre-existing evaluation bounds 



User identity 



Program characteristics 



Intra-organizational features 



External features 



With respect to my own program and the above topic, I need to keep the following in mind: 



Factor Affecting Use: Relevant Information: 



A-5. External features 

(community climate & 
influence; role of other 
agencies) 



Things I Can Do To Influence Use: 
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CHAPTER II 
SETTING THE STAGE: FACTOR CLUSTER A 

Worksheets Al-5 are the first of four groups of worksheets we will be 
filling out together. Let's begin by looking at the first worksheet in 
the Factor Cluster A group (Worksheet Al), which you'll find at the very 
beginning of this chapter. 

Take a look at the first line of the worksheet, labeled "Evaluation 
Topic." What we want you to do here is to identify the major focus for 
the evaluation: that is, the major question or concern about your project 
that you want the evaluation to respond to. Once you've identified the 
appropriate topic, please write it out on the line provided. 

Here are some sample evaluation topics for you to look at in case you 
are having trouble phrasing your own: 

Sample Evaluation Topics 

Is the native-language-first approach to bilingual instruction 
in reading effective? 

What training should be provided to regional sales staff? 

Shall we keep the present Social Studies curriculum or 
change it? 

How can we improve the quality of the examinations in our 
nursing education program? 

Now that you've written out an evaluation topic, you're on your way 
to structuring the prospective evaluation to enhance use. (And you thought 
this would be' difficult, didn't you?) 

Underneath the line where you've written your evaluation topic, you'll 
notice there is a diagram labeled "Factor Cluster A". Remember we said 
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earlier that this workbook would cover only those factors which research 

has shown to be most influential in affecting evaluation use. Factor Cluster 

A represents one specific grouping of these influential factors. 

"But why", you may ask, "are these particular factors grouped 
together?" The reason is that each of the four specific clusters represents 
what might be considered a distinct phase in the evaluation process. 
Factor Cluster A, which we've termed "setting the stage", comprises those 
factors which are generally regarded as "givens" of the particular setting 
within which the evaluation will take place. Although these factors are to 
some extent fixed, you - as organizer of the evaluation - may nevertheless 
be able to influence them in such a way as to promote use. More on the Idea 
of Influence later. For now, we suggest that this first group is the 
most logical and most effective starting point by drawing an arrow at the 

other end of the Factor Cluster diagram ( >). Once we have finished 

our work with Factor Cluster A, this arrow will guide us on to the next 
Factor Cluster. 

But of course we are not nearly-finished with Factor Cluster A yet - - 
we've just begun. Let's look at the individual factors that make up this 
factor cluster. There are five factors listed: 

Pre-existing evaluati on hnunrtg 

User Identity 

Program characteristics 

Intra-orqanizafiona T features 

External features 

These five factors are separated from one another by solid black lines, 
so that we may consider each of them individually. 
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Underneath the factor cluster chart we've provided a reminder phrase 
to jog your memory about your own project's characteristics as you fill out 
the worksheet. 

With respect to my own program and to the above evaluation 
topic, I need to keep the following in mind: 

Underneath the reminder phrase are three major headings: 



FACTOR AFFECTING USE 


RELEVANT INFORMATION 


THINGS I CAN DO TO 






INFLUENCE USE 



We have already filled In the Information In the first column. In each 
of the other columns, you will be filling In your own Information relative 
to the project being evaluated. Let's consider now what each of the column 
headings represents. 

1 FACTOR AFFECTING USE 

"Factor Affecting Use" refers to each of the Individual factors that 
make up the factor cluster. Factor Cluster A Includes five such factors* 
each of which has been given Its own workbook page. Thus, on the page you 
should now be looking at, under the column heading "Factor Affecting Use" 
you will see printed "A1. Pre-existing evaluation bounds". This Is the 
first factor we will examine In detail. 

Underneath the factor heading Itself, we have listed those Individual 
elements which together make up that factor. Earlier 1n our discussion, 
we defined "factor" as "any characteristic or element present In a given 
situation which can affect the outcome of that situation". Now we would 
like to spell out for you exactly what those characteristics or elements are. 
In the case of the factor "Pre-existing evaluation bounds", those character- 
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1 sties/elements are: written requirements, other contractual obligations, 
and fiscal constraints. Let's define these terms. 

The term written requirements refers to any legal or contractual 
obligations that the evaluation Is required to fulfill. Examples of such 
obligations are legal codes, federal/state requirements, Institutional 
operating policies, specific procedural requirements of the unit being 
evaluated, and requirements specified by an RFP. These written requirements 
bind you to producing the specified Information, regardless of whatever 
other Information you need or want the evaluation to produce In addition. 

Let's look at a specific example of written requirements In operation. 
Put yourself once again In the position of the bilingual project director. 
Your project 1s funded by an agency of the federal government, the Office 
of Bilingual Education and Minority Languages Affairs. This external agency 
requires that all evaluations of bilingual projects produce data on student 
achievement In specified grades. Therefore, In order to satisfy the 
requirements of the funding agency, you and your evaluator must produce 
the requested Information In order to fulfill his/her contract. 

Again, let's play the role of the bilingual project director. In 
addition to satisfying the requirements of the federal funding agency, you 
want the evaluator to prepare a special report that you can present to the 
School Board, a report tailored specifically to their concerns about the 
project. You may want to put this requirement in writing so that the 
. evaluator will be contractually bound to produce the report for you. 

The third element of the factor "Pre-existing evaluation bounds" Is 
termed fiscal constraints. This term refers to the restrictions on the 
evaluation Imposed by budgetary considerations. In the real world, no 
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evaluation can do everything we would like It to do; there Is simply not 
enough money. No one has unlimited resources to expend on a single 
evaluation for a single project. Given these fiscal limitations, you will 
need to make choices as you set up your evaluation requirements. 

| RELEVANT INFORMATION j 

Now let's apply what we've been discussing to the evaluation topic 
you've written at the top of the worksheet. The second column Is headed 
"RELEVANT INFORMATION". Think about your specific evaluation topic and 
your specific project. Then think about what you are required to produce 
for your own needs; and about how much money Is available for the 
evaluation. 

After you've thought these questions through and arrived at some answers, 
take your pen or pencil and summarize these answers In column 2 of Worksheet 
Al. To help you organize your thoughts, we've provided a numbered list In 
the blank column. Use as many numbers as you need to write down all the 
relevant Information. You will note that from now on, throughout the rest 
of the workbook, we have provided completed worksheets to serve as samples. 
For purposes of Illustration, we have taken the hypothetical case of a 
school board faced with a decision about a Social Studies curriculum. The 
evaluation topic 1s phrased as follows: "Shall we keep the present Social 
Studies curriculum or change It?" The evaluation 1s Intended to answer 
this question. This topic provides continuity for our sample worksheets, 
just as your own evaluation topic provides the context for your efforts at 
filling In the blank worksheets In each chapter of this workbook. If you're 
having trouble getting started, you should refer to the completed sample 
worksheet below (Figure 2). 
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(Insert Figure 2 here) 

| THINGS I CAN DO TO INFLUENCE USE 

Now that you've filled In the second column, let's turn to the third 
column, headed -THINGS I CAN DO TO INFLUENCE USE". For this column, think 
of any actions that you can perform, any steps that you can take, that will 
Influence a particular factor In such a way as to facilitate use. The 
Intent Is to clarify, explicate, or modify the factor - - In short, to 
"keep" the factor In order to enhance Its potential positive effect or to 
reduce Its potential negative effect on use. 

Here are some questions to ask yourself: 

What can I do to make sure that the evaluator clearly understands 
the written specifications? 

What can I do to keep the evaluation on track so that deadlines 
will be met? 

Who else can I Involve to help assure that the evaluation process 
goes as scheduled? 

Is there any way I can deemphaslze certain contractual obligations 
If they are not highly relevant to the evaluation topic or to the 
concerns of the potential users? 

Is there any way I can use my project staff to provide low-cost or 
no-cost services so that fiscal constraints will be lessened? 

You've probably already considered some possibilities while you were 
thinking about the relevant Information. Remember too that some factors 
are more amenable to your Influence than others. Now, take your pen or 
pencil and fill In the third column of Worksheet A1. Make sure that you 
phrase your statements In terms of specific actions; list only things 
you can do. 
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If you need help, refer to Figure 3 below - - a completed sample 
worksheet. (By the way, notice as you are filling out the worksheet that 
there may or may not be a one-to-one correspondence between an Item In the 
"RELEVANT INFORMATION" column and an Item In the "THINGS I CAN DO TO INFLUENCE 
USE" column). 

(Insert Figure 3) 

You should now have one completely fllled-out worksheet. You've 
already Identified several things which you can do toward organizing your 
evaluation for use. These may seem like small .steps, but we've only just 
begun the process with the first component of the first factor cluster. 
As we proceed, you will begin to see the Interplay among various factors. 

Turn now to the next worksheet for Factor Cluster A, labeled "Worksheet 
A2". This worksheet deals with the second of the five factors In Factor 
Cluster A, labeled "User Identity". 

(Insert Diagram 1) 

Once again, we have preprinted for you the Individual elements which together 
make up the factor "User Identity". They are: range, organizational 
positions, and professional experience levels. Let's define these terms. 

By range , we mean the entire spectrum of potential evaluation users. 
Often an evaluation has many potential or groups of users. As the sample 
worksheet shows, the evaluation of the social studies curriculum has at 
least three potential users: yourself, as administrator; the Board of 
Education; and the Social Studies subcommittee. Let your Imagination 
roam free when envisioning who the potential users nay be - - It Is better 
to be overly comprehensive early on than to realize belatedly that you 
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have left out some Important group. 

By organizational position , we recognize that each of the potential 
users or groups of users may occupy different niches In the organizational 
structure. Since the Interrelationships among these different organizational 
positions may affect how you choose to Interact with each user, It Is 
Important that you recognize these Interrelationships. It Is also Important 
that you Identify early on the key Individuals within each potential user 
group. Those key Individuals can assist your efforts to make use happen 
by serving as enthusiastic and committed role models for the others. 

The term professional experience levels means something different. It 
refers not to the organizational levels of potential users, but rather to 
their length of service and breadth of experience In their jobs. Knowing 
how your potential users differ on this characteristic can help you to see 
that the evaluation Information and reporting techniques are properly 
focused. Again, Identifying the key Individuals - those with the most 
Influence on the decision making process - within your organization or 
project Is Important for guiding the Interactions between yourself, the 
evaluator, and those Individuals. 

Let's Illustrate these concepts. Suppose that you are the vice- 
president In charge of sales operations. You are trying to decide whether 
to hold workshops where regional personnel can Improve their sales 
techniques. In thinking about the factor, "User Identity", you should be 
asking question? like the following: Who Is most likely to get Involved 
In these workshops either as participants or as presenters? Who else 
has an Interest In the outcomes of the workshops? What positions do these 
Individuals occupy within the company structure? (Should both supervisors 
and subordinates be invited? Are all supervisors on the same organizational 
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level?) Which of the potential participants hold key positions within 
the organization? How have these particular Individuals worked together 
In the past? How experienced Is each of these Individuals In his/her 
job? 

Turning from this general example to your specific evaluation topic, 
you should now take a few moments to think along the same lines. 

Who Is a likely user of the evaluation Information? 

What are these people like? 

How and where do they fit In you organization? 

How do they Interrelate? 

How experienced are they? 

When you are satisfied that you have answered these questions In your own 
mind, take your pen or pencil and fill In the column headed "RELEVANT 
INFORMATION" on Worksheet A2 at the beginning of this chapter. If you need 
further guidance, refer to the completed sample worksheet below (Figure 4). 

(Insert Figure 4) 

When you are satisfied with your list In column 2, turn to the third 
column, headed "THINGS I CAN DO TO INFLUENCE USE". Remember that here you 
want to list specific actions you can take to Increase the likelihood that 
evaluation Information will be used. For example, Is there anything you 
can do to target the specific users you most want to reach? Or do you want 
to do some restructuring of your organization for evaluation because of who 
the key Individuals are that you've Identified? Perhaps you should arrange 
for some meetings between the evaluator and certain of your more (or less) 
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experienced potential users. On the other hand, there May be little 
you can do to Influence this factor. That Is, the potential users are 
H g1vens H that are not amenable to change or modification In the sense 
that you see no way to take advantage of their characteristics, positions, 
Interrelationships, or experience so as to promote use. Make sure that 
you exhaust all reasonable possibilities, however, before you reach this 
conclusion. 

When ycjj are satisfied with your mental examination of these questions, 
fill In column 3 on the worksheet. You may refer to the completed sample 
worksheet (Figure 5) If you need more assistance. 

(Insert Figure 5) 

By this point, you are probably becoming aware of both the complexity 
of these factors and the Interrelationships emerging among them. For 
convenience, we have divided the relevant factors Into four general factor 
clusters., each of which represents a different stage In the evaluation 
process. In reality, however, these factors can overlap at many points 
1n the process. Indeed, It Is often difficult to separate one factor from 
another as the evaluation progresses. He ask you here to try to keep them 
separate only so that you can be sure of Identifying all the elements opera- 
ting within your own evaluation situation. In the discussions which follow, 
we will point out some of these areas of Interrelationship and overlap. 

(Insert Diagram 2) 

Now we are ready to examine the third of the five factors In Factor Cluster 
A: "Project characteristics". Once again, you will find preprinted for 
you on Worksheet A3 the Individual elements which make up this factor. 
They are: age, maturity, Innovatlveness, and overlap with other projects. 
Let's define each of these terms. 37 
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By age, we nun simply how long the project has been In existence. 
A new project his different evaluation needs end questions then one which 
has been functioning for many years. Haturlty refers to the relative 
permanence of project features; that Is, Is the project still In a state 
of development, or have most of Its operations been fixed by now? 

The Innovatlveness of the project reflects the degree to which It 
embodies something new, something creative, something unique, or perhaps 
something which differs from the unit's usual types of activities. 

Overlap with other projects Is the extent to which the project Is 
part of a larger effort: e.g., one of several similar projects within a 
given system. The term can also refer to the nature of the relationship 
between the project being evaluated and other projects run by the same unit 
or organization. 

These few Individual elements have been selected end grouped Into one 
fector In order to suggest that the more closely the evaluation processes 
and findings ere relevant to the needs and the developmental status of your 
particular project, the greater will be the potential for use. 

Turning now from our general discussion of "Project characteristics", 
let us consider for a moment the project which you nave selected for 
evaluation. Ask yourself the following: 

What Is this project re?11y like? 

How long has 1t been In operation? 

Are project operations/activities still changing? 

Does this project heve special features that Mark it as unique or as 
significantly different from other projects for which you are responsible? 

How does this project "fit" with other projects In the organization? 
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When you feel you have sufflcltntly explored these questions, tike 
your pen or pencil end fill In the column headed "RELEVANT INFORMATION" 
on Worksheet A3. As usual , we have provided a sample worksheet for your 
reference ( figure 6.) 

(Insert Figure 6) 

You will see that we have added some notes In this col urn under the 
heading "Other Thoughts". This Is a way of keeping track of thoughts that 
occur to you as you reflect on a particular factor, thoughts which way 
not be relevant to that factor but which nonetheless you don't want to 
overlook. Make sure that you periodically go back and review these 
"other thoughts" so that you can Include the* on the appropriate worksheet. 

When you are satisfied that you have written down all relevant Information, 
as well as whatever "stray" Information comes to mind that you don't want 
to forget, then proceed to the third column of Worksheet A3, labeled 
"THINGS I CAN DO TO INFLUENCE USE". We're sure you remember that this Is 
where you should focus on specific actions you can take to Influence the 
evaluation situation In favor of potential use. 

Take a few moments now to reflect back on your own .project ,^and .then 
fill out the last column In Worksheet A3. You should note that sometimes 
the characteristics of a project cannot be modified ir. iny significant 
way. It Is all right to decide that In your own evaluation situation, there 
Is little you can do to Influence a particular group of factors. If this 
Is the case, don't worry about not having much Information to fill out In 
the third column of the worksheet. To demonstrate that occasionally, on some 
factors, one may not be abl# to take actions to Increase the likelihood of 
use, we have left column 3 of Figure 7 blank. 
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(Insert Figure 7) 
(Insert Diagram 3) 

The fourth of the Individual factors In Factor Cluster A, termed 
"Intra-organlzatlonal features" , comprises the structural characteristics 
of the organizational unit to which the project being evaluated belongs. 
For example, soma projects are relatively autonomous In their organizational 
structure and lines of authority, while others must clear their activities 
and decisions with a central office or soma other superior body within 
the organization. Moreover, even In those cases where the project reports 
to a higher authority within *he organization, the degree of control Imposed 
by that higher authority may vary. Thus, a central office may give 
considerable leeway to one project under Its jurisdiction, while keeping 
other projects on a tight' rein. 

It Is Important that your evaluator know the exact nature of the ' 
relationship between your project and any higher authority to which It 
reports. Nike sure that you specify just how sensitive and accommodating 
this higher authority is likely to be toward the evaluation needs of your 
particular project. 

An additional complication arises when the evaluator Is a member of 
the central office staff and Is "assigned" to your particular project. 
Depending on this person's credibility with your staff, you may have to 
pave the way If the evaluator Is to Interact efficiently with staff members. 
If you are free to engage whatever evaluator you wish, then you should mike 
sure that your evaluator can Interact appropriately with any central- authority 
to which your project is responsible. The Interrelationships between a 
given project and any higher authority within the organization can have a 
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strong Influence on how the evaluation 1s conducted and, ultimately, on 
how and whether it is used. 

Another complicating element 1s that other kinds of information, 
beyond that produced by the evaluation itself, may be operating 1n your 
evaluation situation. This notion deserves a more detailed explanation. 
We all know that a project staff normally has access to a variety of 
information sources. For example, a staff member may have a personal - 
relationship with a member of the central office staff and, through this 
contact, may find out information which is not public knowledge. This 
private information can, in turn, influence the staff member's attitude 
toward the evaluation or the evaluator. Office "gossip" networks are another 
example of the kind of informal information sources we are referring to. 
Often, the evaluator 1s unaware of the existence of these sources. (Indeed, 
even you may be unaware of them.) The point is that such extra-evaluation 
information sources do exist, even though you may not be able to do much 
about them beyond recognizing the probability of their existence. 

Finally, this fourth factor in Factor Cluster A includes the element 
of the perceived institutional and financial risk of the evaluation. To the 
extent that these perceived risks and costs outweigh the perceived benefits 
of the evaluation, the potential for evaluation use is reduced. For example, 
your organization may initially fear that a negative evaluation report will 
result in a loss of funds, a loss of jobs, or some other dire outcome. 
Or your organization may feel that money spent for the evaluation might better 
be spent on other project activities, that the information to be gained from 
the evaluation is. not worth the cost. Such attitudes can negatively affect 
how much the evaluation will likely be able to accomplish in terms of your 
own project concerns and questions. 
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Turning now from our general discussion of "intra-organlzational 
features", let's consider your project* Review 1n your own mind how your 
project fits Into the' overall administrative structure o* your organization 
(perhaps you are in charge of an autonomous unit, whic oes not have to 
report to a higher authority); try to Identify what private or "behind-the- 
scenes" information sources your staff members are likely to tap during 
the evaluation; and consider what types of institutional and financial risks 
are Involved in the evaluation. When you feel that your thoughts are 
sufficiently detailed, then turn to Worksheet A4 at the beginning of this . 
chapter and fill in the column labeled "RELEVANT INFORMATION". As always, 
you may refer to the completed sample worksheet (Figure 8) 1f you need to. 

(insert Figure 8) 

« 

Now let's consider what you might do to Influence some of the circum- 
stances which you've just described. For Instance, if you are the vice- 
president for sales, you are probably used to dealing with some higher auth- 
ority 1n your organization. Now is the time to use whatever tactics 
have worked best for you 1n the past in looking out for the interests of your 
project. Now is also a good time to use whatever persuasive powers you 
have to convince your staff (or your higher organizational authority) that 
the evaluation entails more benefits than risks, financial or otherwise. 
Of course, in order to be convincing, you must believe this yourself 1 
And last but not least, now 1s a good time to step back and take stock of 
what Information sources are generally tapped Into by your staff members. 
Perhaps some trusted subordinates can clue you in on these informal networks, 
so that you will be forewarned of any negative information that might 
adversely affect the conduct and the outcome of the evaluation. And don't 
forget how important 1t is to share what you have learned with your evaluator; 
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otherwise, you may be undermining the potential for evaluation use. 

When you have carefully examined all these issues, take your pen or 
pencil and fill out the third column of Worksheet A4^ "THINGS I CAN DO TO 
INFLUENCE USE", we have completed our worksheet below (Figure 9). 

(insert Figure 9) 

Now, let's see where we are 1n our discussion of Factor .Cluster A. 
We've just dealt with the fourth of the five individual factors making up 
the cluster. Now we come to the fifth and last factor. 

(insert Figure 4) 

The final factor 1n the cluster, "External features", comprises three 

interrelated elements: community climate, community influence, and the 

role of other agencies. Let's discuss each of these in turn. 

The co rnnunity climate refers to the atmosphere typifying relations 
between a given project and the larger community in which that project 
operates, For example, a sports program at a local YMCA has a certain 
relationship w?th the larger community of which it is a part. Community 
climate is especially important when a project requires formal or informal 
community support (particularly of a financial nature) - - or, at the very 
least, community neutrality — in order to operate. (As an example of the 
latter situation, certain types of medical programs, such as Planned 
Parenthood, can probably not operate if the community is actively opposed 
to them.) Some programs, however, are strictly internal operations within 
a larger organization: since they do not interact directly with the larger 
community, community climate is an irrelevant consideration. You, as the 
responsible organizer for the evaluation, will need to determine whether and 
how your project relates to the larger community. 
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The second element of this factor Is community Influence. The extent 
to which the community may try to exert Its Influence depends In large 
part on Its perceived stake In the project being evaluated. For Instance, 
the community may take an active Interest In after-school sports programs, 
particularly baseball and football, and may thus see Itself as having a 
stake in the evaluation of social programs. (The same 1s true of community 
climate). On the other hand, neither community climate nor community 
Influence Is a very Important consideration In the evaluation of most 
private-sector programs. For example, the community at large probably 
takes little Interest In a bank's plans to expand Its computer training 
project for bank managers and Is thus Indifferent to any evaluation of 
that project. 

The Important point to remember Is that when a community has an expressed 
Interest or stake In a project, Its Involvement In the evaluation Increases 
the potential for Information use. 

The role of other agencies refers to the continuing role that an external 
agency may play during the course of the evaluation. As we have seen, an 
external agency can exert Its Influence before the evaluation begins through 
Its written requirements that the evaluation must satisfy; here, we suggest 
that some agencies may continue to affect the evaluation In several ways: 
for Instance, by requiring that certain methods or procedures be followed, 
or by changing its written requirements In mid-stream, thereby, necessitating 
changes In the amount and type of data to be collected. 

To the extent that any changes suggested or required by an external 
agency threaten to have a negative Impact on the potential for use, It 
behooves you, as responsible administrator, to do what you can to make the 
agency reconsider Its position. At the very least, you should try to 
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minimize problems by being aware of the potential role of other agencies. 
Of course, we are not overlooking the fact that many projects do not have 
to worry that external agencies may affect the conduct of their evaluations. 
Only you can determine, as you progress through this workbook, which of the 
factors described apply to your own project. 



An Example 



Now let's Illustrate how this final factor In Factor 
Cluster A - External features - can affect an evaluation's 
potential for use. Think back, If you will, to the example 
of the bilingual project which uses a native-language-first 
approach to reading Instruction. For a project such as this, 
external features assume an Important role for the evaluation 
organizer. For example, the community climate surrounding the 
project Is critical to the project's success. If parents and 
others are opposed to this Instructional method, they are 
likely to communicate these attitudes to their children, who 
may react negatively to their classroom experiences, thereby 
reducing the project's chances for success. In addition, the 
community can exert its Influence by refusing to support the 
project's efforts to get continued funding from the government 
and by criticizing the project at community and school board 
meetings. Further, the government (an external agency In this 
* context) not only imposes requirements for the type of Infor- 
mation to be produced by the evaluation but also suggests the 
methods to be used in obtaining certain data. If you are the 
administrator responsible for organizing this evaluation, you 
will certainly need to take whatever measures you can to deal 
with these elements. Otherwise, you will jeopardize the chances 
of the evaluation's being used by these audiences. 



It's time now to think specifically about your own project In relation 
to the three external features: community climate, community Influence, 
and the role of external agencies. When you have sufficiently gathered 
your thoughts, turn to Worksheet A5 at the beginning of this chapter and 
fill In the second column, "RELEVANT INFORMATION" . If you need help, 
refer to the completed sample worksheet (Figure 10). 



(Insert Figure 10) 



When you ar« satisfied with what you've written, start thinking about 
what you might do with respect to these elements in order to promote the 
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likelihood of evaluation use. 
An Example 

If you were the director of the bilingual project, here are some of the 
things you might do: 

(a) Hold community meetings to explain carefully and In detail 
what the bilingual project Is trying to accomplish and how; 

(b) Set up a system of Informal Informational meetings for parents, 
staffed by other parents favorable to the project; 

(c) Cultivate positive relationships with media representatives 
(from the local newspaper, radio, and television) and try to get 
favorable publicity when possible; 

(d) Involve parents as teacher aides; 

(e) Set up parent workshops to make learning games, materials, etc., 
parents can use at home with their children; 

(f) Identify major community concerns, and make sure the evaluation 
addresses those concerns. 

Now think again about your own project: what strategies can you put Into 
practice to deal with the concerns you've raised In the "RELEVANT INFORMATION" 
column? When you have Identified your approach, list these Items In the third 
column, "THINGS I CAN DO TO INFLUENCE USE", as we have done on the sample 
worksheet (Figure 11). 

(Insert Figure 11) 

RECAP : Now that you've completed all five worksheets for Factor Cluster A, 
let's take a moment to review what you've accomplished thus far. You've 
been doing some serious, focused thinking about the project for which you 
have administrative responsibility. In your thinking, you've begun to set 
the stage for getting the evaluation off to a good start, one which Is 
more likely to result In having the evaluation Information used. How have 
you done this? By filling out Worksheets A1-5, you've delineated the basic 
characteristics of the project, you've Identified the people most likely 
to be users of the evaluation fnfonaatlon to be produced, you've noted the 
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limits set on the evaluation In terns of specific Information requirements, 
and you've described the organizational and external features that may 
affect evaluation use. All this Is no small accomplishment! Further, 
you've begun to make notes on additional Information which may affect 
the conduct of the evaluation. Finally, you've set some specific 
tasks for yourself by listing steps you can take to promote the likelihood 
that the evaluation will make a difference. 

Before you are tempted, however, to rest on these not-Inconsiderable 
laurels, remember that we have worked through only one of the four factor 
clusters. Perhaps It's time now to capitalize on your sense of accomplishment 
and your enthusiasm, and continue on to Factor Cluster B. You'll remember 
that all the worksheets for Factor Cluster B are provided at the beginning 
of the chapter. Turn the page, and you'll find Worksheet Bl awaiting 
your attention. 

(Inserts worksheets Bl-5 on separate left-hand pages) 
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Factor Affecting Use: Relevant Information: 



Things I Can Do To Influence Use: 



A-1. Pre-existing 
evaluation bounds 

(written requirements; 
other contractual obliga- 
tions; fiscal constraints) 



amid 3hM> IWB 



2. AcUoo^ia^ibJ^maii J^Jt|bt.( 2. 



Mai eudualuvu ■ 



Aiido Jb-okul^ alt othtA/itaAtbtc 
cu/vucuiou 



5. 



ERIC 



48 



Organizing For Evaluation Use 



Figure. 3 
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Intra-organizational features 



External features 



With respect to my own program and the above topic, I need to keep the following in mind: 



Factor Affecting Use: 



A-1. Pre-existing 
evaluation bounds 

(written requirements; 
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Organizing For Evaluation Usa 

Worksheet A2 



Figure 5 
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Worksheet A4 



Figure 9 
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Figure 10 
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Worksheet A5 



Figure II 
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III. IDENTIFYING/ORGANIZING THE PARTICIPANTS: 
FACTOR CLUSTER B 
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Worksheet B1 



Evaluation Topic- 



Factor Cluster B: 



User interest in evaluation 



User commitment to use 



Evaluator characteristics (selected) 



Evaluation procedures— plan 



User professional style(s) 
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Evaluation Topic: 



Factor Cluster B: 



User interest in evaluation 



User commitment to use 



Evaluator characteristics (selected) 



Evaluation procedures— plan 



User professional style(s) 
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With respect to my own program and the above topic, I need to keep the following in mind: 
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Wostefoeet B3 



Evaluation Topic: 



Factor Cluster B: 



User interest in evaluation 



User commitment to use 



Evaluator characteristics (selected) 



Evaluation procedures— plan 



User professional style(s) 



With respect to my own program and the.above topic, I need to keep the following in mind: 
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CHAPTER III 

IDENTIFYING AND ORGANIZING THE PARTICIPANTS: FACTOR CLUSTER B 

The last chapter discussed those factors (Factor Cluster A) which "set 
the stage" on which the evaluation unfolds, the "context" factors which pro- 
vide the framework within which the evaluation 1s conducted. In discussing 
Factor CI uster B, we will be concentrating more heavily on the human factors 
which play an Important part 1n the evaluation use process. Thus, Factor 
Cluster B Involves Identifying and organizing the evaluation's participants. 
This cluster comprises five factors, as Indicated below: 
(Insert Factor Cluster B) 

Again, these factors are separated from one another so that we may 
consider each of them Individually. Since the third factor subsumes several 
subheadings, a brief explanation 1s" 1n Order. 

In the case of the factor labeled "Evaluator characteristics," not all 
Its possible component elements have been shown by research to be directly 
related to use. Thus, we have singled out for your attention only those 
elements which are relevant. These selected elements are reproduced 1n Italic 
type 1n the Illustrative charts; 1n our listing of the factor, we remind you 
that not all elements are relevant by using the word "selected" 1n parentheses. 
(Insert Diagram 5) 

Now let's return to Factor Cluster B Itself and examine what each factor 
represents. The first 1s termed "User Interest 1n evaluation." This factor 
pertains specifically to the potential users of the evaluation Information. 
Thus, 1t 1s considered a "human factor" (to recall our earlier discussion 1n 
Chapter I of this workbook). Included 1n this factor are the following 

* 

elements: the potential users' views about the project to be evaluated; 
their questions and concerns about the project; their basic orientation toward 
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Evaluation Topic: 



Factor Cluster B: 



User interest in evaluation 



User commitment to use 



Evaluator characteristics (selected) 



Evaluation procedures— plan 



User professional styjejs) 



With respect to my own program and the above topic, I need to keep the following in mind: 



Factor Affecting Use: Relevant Information: 
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Wosfeteet B4 



Evaluation Topic: 



Factor Cluster B: 



User interest in evaluation 



User commitment to use 



Evaluator characteristics (selected) 



Evaluation procedures— plan 



User professional style(s) 



With respect to my own program and the above topic, I need to keep the following in mind: 
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evaluation In general and toward the evaluator In particular; their expecta- 
tion about what the evaluation can accomplish; and finally, their perceptions 
of the risks entailed In the evaluation. 

As you can see, this factor Is complex and covers a diversity of topics. 
The unifying thread 1s the user; that Is, the factor Is concerned only with 
the orientations and concerns of potential users of the evaluation. Of course, 
any evaluation Is likely to have several potential users or groups of users, 
whose orientations and concerns may differ, thus Introducing some complications. 
We will approach our discussion, therefore, from the users' viewpoint. 

The term views about the project to be evaluated refers to the beliefs 
that users hold before the evaluation actually begins. For example, some 
people already have firm views about the project. A possible difficulty 
arises when these views are so strong that they cannot be changed, regardless 
of what the evaluation shows. You will need to be sensitive enough to this 
possible problem to alert the evaluator to It during one of your early briefings. 

As we just pointed out, whatever your project, there are likely to be 
at least several potential users of the Information produced by the evaluation. 
Each of these potential users Is also likely to have his/her own set of parti- 
cular questions or concerns about the project and which the evaluation might 
address. Sometimes It may be Important that you, as responsible administrator, 
set some priorities as to which users' questions will take precedence. This 
type of decision Is especially likely to be necessary when the funds and/or 
the time available for the evaluation are limited. 

The questions and concerns raised In connection with the project under 
evaluation may have differing degrees of urgency attached to them. Knowing 

"... 'A 

which questions should'be addressed first can help you to guide your evaluator 
In structuring the evaluation so that the most urgent questions get taken care of. 
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Potential users can also differ on the dimension termed "orientation 
toward the evaluation and toward the evaluator." Their orientation may be 
negative, positive, or neutral. Such differences In orientation can arise 
from several sources. Those with negative attitudes may have been "burned" 
by a previous evaluation: perhaps they feel that it consumed too much time 
and produced too few benefits, or perhaps they found the demeanor or behavior 
of the evaluator offensive. Those with positive attitudes may have had very 
good experiences with a past evaluation and so may welcome more of the same. 
Those whose orientation Is neutral may simply adopt a "let's wait and see" 
stance. Whatever the case, the evaluator can use your help In getting a sense 
of the ballpark In which he/she 1s likely to be playing. 

As the above discussion Implies, potential users are also likely to 
differ In their expectations about what the evaluation can accomplish , the- 
next element of the "User Interest" factor. Some of these divergent views may 
be logical developments of the users' perceptions of the need for the evalua- 
tion. You should communicate Information on these expectations to your 
evaluator, so that he/she can be as effective as possible when Interacting with 
potential users. 

The final element In this factor, termed perceived risks of the evaluation , 
can be very Important In the evaluator-user relationship. Potential users may 
believe that an evaluation carries some risk, be It as drastic as job loss or 
as minor as a small program modification. Whatever the case, the person who 
feels threatened In some way by the evaluation will probably resist It. You 
can help defuse this potentially damaging situation by bringing these perceptions 
of risk out Into the open, through early and frank discussion with both the 
users and the evaluator. Your comments may help to remove some of the threat 
of the evaluation. 
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We've now given you a filr amount of Information about the first of the 
five factors In Factor Clusters. While It's fresh In your mind, you should 
fill out Worksheet Bl at the beginning of this chapter. 

As usual. Worksheet Bl reproduces the entire factor cluster plus the 
reminder phrase to jog your memory about your own project's characteristics 
as you fill out the worksheet. The first column, "FACTOR AFFECTING USE," 
has already been filled out for you with the terms used, to describe each of 
the elements of the factor "User Interest In evaluation." Keep these elements 
In mind as you look at the second column, "RELEVANT INFORMATION." Now think 
about the evaluation topic which you've written at the top of the . 
worksheet ; think about all of your potential audiences In terms of their 
attitudes and expectations, their questions and concerns about the project 
and their perceptions of the risks of the evaluation. 

When you feel that you have adequately explored these topics In your own 
mind, take your pen or pencil and fill In the second column on the worksheet. 
You may refer to Figure 12 for assistance If you wish. 

(Insert Figure 12) 

When you are satisfied that you have noted all essential Information, as 
well as any other thoughts you may have that you don't want to forget, then go 
on to the third column of Worksheet Bl. Here again, you should focus on what 
you can actually do to Influence the evaluation situation In favor of use. 
Take some time to reflect back on your own evaluate situation, and then fill 
out the last column In Worksheet Bl. Our completed sample worksheet Is shown 
In Figure 13. 

(Insert Figure 13) 
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By now, you my be noticing that some of the Items you write down In the 
"THINGS I CAN DO TO INFLUENCE USE" column tend to be repeated from one worksheet 
to another. Don't worry about this repetition. It simply confirms what we 
suggested earlier: that It Is difficult to separate Into discrete elements 
what 1s essentially an Interactive process. You might want, at this point, 
to begin making a separate list of topics to talk to your evaluator about early 
on In the course of evaluation, so that you can eliminate any duplications In- • 
your listing as you proceed through the remainder of the worksheets. 

Have you filled out Worksheet Bl? 

(Insert Diagram 6) 

Now let's turn to the .second of the factors In Factor Cluster B, termed 
"User commitment to use." Just how committed are those persons whom you have 
Identified as potential users to Implementing the findings of the evaluation? 
To some extent, the commitment of any given Individual or group depends upon - 
their previous experience with evaluation and their general attitudes toward 
evaluation. Perhaps one of the potential users — say, a project staff member — 
looks upon the evaluation as a new challenge, one which can provide feedback that 
will help him or her to Improve job performance. In this case, you can congratu- 
late yourself on already having a user committed to use. Such a situation makes 
your task or organizing for evaluation use that much. easier. . Since we live In 
an Imperfect world, however, the likelihood 1s that you will have one or more 
potential users who are at best Indifferent, and at worst openly negative 
toward the evaluation. Once you have analyzed where various potential users 
stand on the continuum of commitnent toward use, you should then think about 
what you can do to turn around some of the people lass favorably disposed toward 
use and to capitalize on the positive commitment of others. 
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Since we don't want to belabor the point about the importance of commit- 
ment to use, we'll ask you now to consider where your own potential users 
stand on this continuum. Make sure you review in your mind all of the potential 
users before you fill out the second column of Worksheet B2, "RELEVANT INFORMA- 
TION," as we have done below in Figure 14. 

(Insert Figure 14) 

Now that you've identified where your users stand on the continuum, perhaps 
you see that your work is cut out for you! Or perhaps you are fortunate enough 
to have a group of users who stand ready to deal constructively with the evalua- 
tion information produced. 

Consider now what you can do with all groups of users. How can you 
capitalize on the positive orientation of those eagerly awaiting the evaluation, 
process? What can you do to reorient those who wish that the evaluation would 
go quietly away and leave them alone? How can you convince those who feel that 
the outcomes of the evaluation really won't affect them?" How can you motivate 
those who are sitting on the fence, undecided about which position to take? Now 
is the time to play the master strategist, the psychologist, the leader. After 
all, you are in charge of the project; it's up to you to set the example, to let 
people know what you expect from them, and above all, to motivate your staff as 
they become increasingly involved in the evaluation process. 

Now that you are fired up, put your action thoughts down on paper by 
filling out the final column of Worksheet B2, "THINGS I CAN DO TO INFLUENCE USE." 
We've gone ahead and completed ours in Figure 15. 

(Insert Figure 15) 
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(Insert Diagram 7) 

We've now arrived at the third component of Factor Cluster B, "Evaluator 
characteristics," one of the two most Important factors to consider as you try 
to maximize the potential for evaluation use. When you stop to think of it, 
the two most important people 1n the entire evaluation process are yourself, 
as the person responsible for organizing and approving the evaluation, and the 
evaluator, who actually carries out the evaluation activities. It 1s essential 
to evaluation use that both of you be firmly committed to making use happen. 
Therefore, the choice of an evaluator deserves your most serious consideration. 
As we pointed our earlier, however, sometimes you simply do not have a choice 
In the matter of who your evaluator will be. This 1s often the case in large 
government units or corporations, which have their own 1n-house evaluation units. 
There, evaluators are generally assigned on the basis of availability to 
perform project evaluations within the organization. 

Perhaps your organization has Its own 1n-house unit responsible for research, 
evaluation, and development activities. If so, the likelihood of your being able 
to select your own evaluator diminishes considerably, though you may have some 
say over which of several evaluators within the unit 1s assigned to your project. 

On the other hand, you may be running a project which has free rein in its 
choice of an evaluator. Perhaps one of your" own staff can assume the role of 
evaluator for a period of time, or even for a given portion of each work week. 
In this case, it is particularly important that you have clearcut criteria for 
making the best selection possible. Let's talk now about these criteria, which 
are applicable to examining the qualifications of any evaluator. 

The background and other Identifying characteristics of the evaluator are 
Important primarily 1n terms of the potential users' possible reactions 
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to the evaluator. For example, some people may have particular (and idiosyn- 
cratic) preferences regarding an evaluator 's age, gender, race, speech patterns, 
physical attributes, and the like. As unfair as some of these preferences may 
seem to you personally, you nonetheless need to recognize that they can affect 
the ultimate use of the evaluation information. 

A related issue is that of the evaluator' s credibility, both with yourself 
and with your potential users. This element Involves the staff's trust and 
confidence in the evaluator. You may wonder how an evaluator can have credibility 
when he/she 1s an unknown to the users, as 1s the case when you hire a new 
evaluator to conduct an evaluation for your project. The development of credi- 
bility need not wait until the evaluator actually begins Interacting with your 
staff: perhaps his/her reputation precedes the actual hiring; perhaps you have* 
circulated the person's resume, samples of work, or other supporting documents 
which serve to Introduce the evaluator prior to the start of work; or perhaps 
the evaluator has done work for acquaintances of your staff. An evaluator who 
has previously worked with your project, on the other hand, 1s a known quantity. 
If the evaluator's credibility 1s high, then the evaluation. will likely begin on 
a positive note; if the credibility 1s somewhat damaged by prior contacts, then 
you need to engage 1n some damage control to minimize the negative effects of 
diminished credibility before the evaluation actually begins. 

Credibility, however, 1s not static; it is constantly changing during the 
course of an evaluation. We hope that such change is for the better. But you 
also need to recognize the unhappy possibility that despite your best efforts 
at selecting an appropriate evaluator, circumstances may be such that potential 
users lose faith in the evaluator as the evaluation proceeds. 
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Let's consider how this may happen. Suppose that you are a bank manager 
1n charge of upgrading computer systems in a large, multi-office division. 
You are allowed by higher management to select your own evaluator. After 
interviewing numerous candidates, you choose one who seems to have all the 
"right stuff" for the job. Prospects are high for ending up with a truly 
useful evaluation. Two months into the job, however, something unforeseen 
happens: the complex computer system which is being installed finally outstrips 
the evaluator' s computer knowledge, and his ignorance soon becomes evident as 
he deals increasingly with computer experts. Reports start filtering back to 
you that the evaluator has lost credibility. Staff members and other potential 
users, perceiving his lack of technical expertise, no longer have faith in the 
evaluation he is conducting. 

In other instances, diminished credibility may be the result of offensive 
personality traits., loss of rapport with the staff, the quality of the evalua- 
tion report, or other similar problems. 

Sometimes you may have to make tradeoffs in your choice of an evaluator. 
Should you select the person with great technical competence but poor interpersonal 
skills and little sensitivity to the goals of the project? Or should you go with 
the person who interacts well with the staff, understands exactly what the 
project is trying to accomplish, but is not as technically capable as you would 
like? Only you can make such a decision, based on your own experience, feelings, 
and needs. 

Of course, the amount of interaction the evaluator will have with your 
staff depends on the evaluator's choice of a particular role and his/her 
orientation toward user involvement. By choice of a particular role we mean 
the stance the evaluator takes toward the project staff, the potential users, 
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and the project Itself. For example, one evaluator may choose the role of 
detached and objective scientific Investigator; such an evaluator will be 
concerned with setting up ideal experimental conditions, constructing a valid 
research design, and maintaining some distance from the project staff. 
Another evaluator may choose the role of colleague, and will emphasize improving 
the project's effectiveness and answering staff and user questions about the 
project, regardless of whether this information amounts to a .research study 1n. 
the classic sense. A third evaluator may choose the role of neutral third 
party, trying to provide helpful information while at the same time maintaining 
some distance from the staff and the project itself. 

The role which your evaluator adopts when dealing with the project is 
closely related to the evaluator's willingness to involve the user. By 
now you are aware that we favor user involvementj to the greatest extent 
possible, in both the planning and the execution of the evaluation. You should 
examine your evaluator candidates carefully on this issue; it is critical to 
the goal of evaluation use. 

When you put together the picture of an evaluator whose personal demeanor 
is acceptable to all parties, who has credibility both with yourself and with 
other potential users, who chooses a role that is colleaguial and consultative 
rather than detached or dictatorial, and who genuinely wants to work with the 
evaluation's potential users, then you have an evaluator who is genuinely 
committed to use. This is the ideal against which you should measure candi- 
dates for the job of evaluator. 

As a final touch to this picture, the evaluator should have political 
sensitivity , if he or she is to function effectively in your environment. That 
is, the evaluator should be able to deal appropriately with persons at all 
levels within your organization; should not commit faux pas in protocol when 
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Interacting with administrators; should understand enough about the constraints 
on your project to formulate recommendations which are practical, reasonable, 
and consonant with the goals of your organization; and should generally be 
aware of what's going on. After all, no real good can come from the evaluation 
if your evaluator alienates k§y individuals or makes recommendations that are 
clearly beyond the reach of your organization. 

Well, we've painted a rather idealized portrait of the use-oriented 
evaluator. Perhaps you are wondering what you do in a real-world situation 
where either you have no choice of evaluator or all your candidates for the 
position are clearly lacking in one important quality or another. Don't just 
throw up your hands and resign yourself to getting along with a second-rate 
evaluator. Instead, stop and consider whether there is something you can do 
to improve the situation. Let's look at the possibilities. 

First, suppose you're saddled with an assigned evaluator who you know is 
not the ideal choice for the position. (Of course, not all assigned evaluators 
are ipso facto of poor quality — and you may be lucky enough to have a good 
one.) Or, you may have even chosen this evaluator .from a less than ideal 
candidate pool. Once you know who this evaluator will-be, you should try to 
get a clear picture of his/her strengths and .deficiencies* especially in relation 
to the work you want performed for your project. Then you can focus specifically 
c on how to correct or bypass these deficiencies in such a way as to promote 
evaluation use. 

Take, for example, the case of an evaluator who lacks political sensitivity. 
Perhaps you could take the person aside and candidly explain the facts of your 
project's life. Let the evaluator know how you expect him/her to act within 
your organization. This is, after all, your prerogative as responsible adminis- 
trator. Or, if the evaluator lacks credibility, there are two ways to deal 
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with the problem. The first applies to those situations where the evaluator's 
lack of credibility seems undeserved. In such cases, you can work specifically 
to raise his/her credibility level: by circulating examples of good work 
the evaluator has done, by expressing your conviction that the evaluator will 
Indeed be able to handle the tasks demanded of him/her, by having the evaluator 
appear personally before a gathering of your staff, or by any other plausible 
method that will give potential users a better opinion of the evaluator.' The 
second way applies to those situations where the evaluator has real deficiencies 
which may Indeed limit his/her ability to do a good job. In such cases, you 
will have to devote a great deal of time and energy to the problem: by speci- 
fying exactly what the evaluation tasks should be and how they should be 
conducted, and by directing the evaluator every Inch of the way. Obviously, 
such close monitoring will represent a considerable Investment on your part. 
Nonetheless, the fact remains that often you can Improve what appears at first 
glance to be an Impossible situation. 

Consistent with our Initial contention that the evaluator 1s one of the 
most Important determinants of an evaluation's potential for use, we have spent 
considerable time discussing the evaluator's characteristics. 

Now It's time for you to do some work. You need to assess your own 
project evaluator relative to the criteria outlined here. How does your 
evaluator stack up? Does he or she have any personal characteristics that 
might pose a problem? Does he or she genuinely want to Involve all potential 
users 1n the conduct of the evaluation? Does he or she have political savvy? 
Is he or she credible? Is he or she sincerely committed to evaluation use? 

If you are presently 1n a situation where you do not yet have an evaluator 
1n mind for your project, then use this opportunity to reflect on the person who 
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would be Ideal for the job. What characteristics would this person need to 
function most effectively in your project context? Are there any. special 
demands that might be placed upon this person were he/she to be hired to 
serve your project? 

When you are satisfied with this examination of your evaluator's actual 
or ideal characteristics, take your pen or pencil and fill-in the "RELEVANT 
INFORMATION 11 column on Worksheet B3. Our completed sample worksheet will be 
found in Figure 16. 

(Insert Figure 16 here) 

Now, unless you are very lucky, you have probably identified a few areas 
of concern about your actual or prospective evaluator. Take a few moments 
to review what you might do to make the situation more favorable for evaluation 
use. In our previous discussion of evaluator characteristics, we gave you a 
few hints about possible steps to take. When you have planned your strategy, 
(assuming that you haven't simply given up and decided to cut your losses), 
list these steps in the third column of Worksheet B3 at the beginning of the 
chapter, "THINGS I CAN DO TO INFLUENCE USE." Our worksheet, reflecting a most 
optimistic scenario, is found in Figure 17. 

(Insert Figure 17) 

(Insert Diagram 8) 

The fourth factor in Factor Cluster B is identified as "Evaluation 
procedures - plan". This term refers to the operational plan — devised 
through consultations between you, the evaluator, and other potential users — 
as to what information will be collected during the course of the evaluation 
and how it will be collected. The evaluation plan must be comprehensive 
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so that all major user concerns are dealt with; appropriate, so that data 
collection is both on target and feasible to carry out; rigorous , so that 
proper methods of data collection are followed; and acceptable to users with 
decisionmaking power, so that any reservations they may- have about using the 
evaluation findings are minimized. 

At this stage of planning your evaluation, you need to make sure that 
you and your users are comfortable with the proposed evaluation activities. 
Any reservations'— for instance, about specific procedures, about the data 
collection instruments to be used, or about the persons to be involved in the 
evaluation — should be worked out to your satisfaction in order to mitigate 
possible negative influences on the use of the evaluation information produced. 
The more you and other users get involved in planning the -evaluation, the more 
likely it is that the evaluation information will eventually be used. So 
this is your opportunity to help ensure that potential users feel like owners 
of, or shareholders in, the evaluation — that they "buy into" the evaluation 
process. Since the evaluation plan will guide all future evaluation activities, 
it is important that it be as good as possible before you formally agree to 
its implementation. 

In devising an evaluation plan, the evaluator has primary responsibility 
for coming up with an initial draft or with recommendations for your considera- 
tion. We do not expect you, as the responsible admini --.'ator, to be an expert 
in evaluation. After all, that is why you have gone to considerable trouble 
either to hire a qualified evaluator of your own or to investigate as best you 
can the qualifications of the evaluator assigned to your project by a central- 
office. But it is expected that you will contribute your knowledge of the 
project, its staff and its operation; your familiarity with the potential 
uiers; and your experience in working with any external agencies that may 
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be Involved with your project. If everyone works together, you should be able 
to come up with an evaluation plan that fits your project's circumstances , a 
plan which will enhance the potential for evaluation use. 

Since we have assigned primary responsibility for developing the plan to 
your evaluator, you might want to jot down, 1n the second column of Worksheet 
B4, any Information that you think you should Impart to your evaluator as he/she 
1s devising the plan. For example, are there particular dimensions you want 
measured, or tests which must be used, or measures which you definitely don't 
want used? Or perhaps you would like the evaluator to make some suggestions 
1n this area. Perhaps, too, there are certain Individuals who should be 
brought Into the evaluation process at specific points along the way; 11st 
their names on the worksheet. If this 1s your first experience with evaluation, 
you may want to note any questions you have about evaluation procedures, or 
any concerns you need to discuss with the evaluator prior to the plan's develop- 
ment* 

You may also want to continue on and fill out the third column, "THINGS 
I CAN DO TO INFLUENCE USE. 11 Perhaps you feel that at this point you don't 
have enough Information about evaluation to fill out this column. That's okay! 
Perhaps you feel that you need to consult with the evaluatdr before defining 
a definite role for yourself 1n the process, and that's okay, too. But 1f 
there 1s some clear action which you can take, go ahead and add that to column 
3. For example, suppose that you want a particular measurement instrument used 
1nthe evaluation. Make a note to yourself in column 3 to give a copy of this 
Instrument to the evaluator as soon as possible. 

We've gone ahead and filled out our sample worksheet 1n Figure 18 below. 
You'll note that the third column; "THINGS I CAN DO TO INFLUENCE USE",has been 
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left blank. In our example, the administrator is not yet sure what evaluation 
procedures are most appropriate to the situation and will need to consult with 
the evaluator before filling in the third column. 

(Insert Figure 18) 

Turn now to the beginning of this chapter and fill out Worksheet B4. 

(Insert Diagram 9) 

The final factor in Factor Cluster B is "User professional style(s)." 
Each of the potential users, including yourself, has certain administrative 
and organizational skills, a certain amount of initiative, and a certain degree 
of openness to new ideas or to change. All of these qualities constitute 
professional style. You must know the potential users well enough to be able 
to capitalize on or, alternatively, to minimize the effects of professional 
style(s) on the evaluation. 

Let's consider how you might make these combinations of qualities work 
for you and your evaluator. Suppose that the project head (not you, of course) 
is known to be a procrastinator. If the evaluation plan calls for data 
collection instruments to be administered at a specific time, then you will 
need to make sure that the project head is on top of the situation. If you 
know ahead of time what pitfalls to avoid, you can eliminate some of the more 
obvious reasons for non-use of evaluation information. On the other hand, a 
project head who has the ability to get things started and to follow through 
on them will probably be a person who actively promotes evaluation use. You 
can probably depend on this individual throughout the course of the evaluation. 

Another situation worth examining in this context is one where the 
evaluator is not your first (or even third) choice for the job. In these 
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circumstances, user professional styles take on an added Importance. A user 
who 1s actively committed to evaluation use (remember Factor B2?) can pick 
up the ball 1f the e valuator drops 1t and can move the evaluation use sequence 
forward on his/her own Initiative. On the other hand, perhaps the e valuator 
1s committed to use. Does this mean you can Ignore the professional styles 
of the potential users? Certainly not! In this Instance, just think how much 
greater the potential for enhancing evaluation use 1s when both the e valuator 
and the users are working on their own Initiative (and perhaps even 1n concert; 
to promote use. 

A final point to consider 1s, just how open to change and to new Ideas 
are you and the other potential users? Any evaluation properly conceived and 
conducted 1s likely to produce unexpected Information about the project and 
Its operation. How willing are you and other potential users to take this 
Information seriously and to make any necessary changes 1n the project suggested 
by the Information? Openness to new Ideas and to change 1s closely related 
to the perceived risks of the evaluation. If users view new Information as 
detrimental to their Interests, then they will probably be unwilling to 
consider 1t seriously or to apply 1t 1n making changes 1n the project. As 
you can see, this factor has Important Implications for evaluation use; lack 
of openness Imposes drastic limitations on potential use. 

Take a moment now to review 1n your own mind the different professional 
styles of the people you have Identified as potential users of the evaluation. 
If they have any characteristics which you feel are either a definite plus or 
a definite minus, 11st them 1n the second column of Worksheet B5. It's not 
necessary to 11st all aspects of user professional styles; Include only 
those which are likely to have a strong effect — either positive or negative — 
on potential for evaluation use. 
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Speaklnn of action, 90 on now to fill out the third column of Worksheet 
B5, "THINGS I CAN DO TO INFLUENCE USE. H Make sure that you consider carefully 
whether there really Isn't something you might do to Improve the potential 
for use before you decide to leave this column blank. As responsible admini- 
strator, you have probably had considerable experience In managing people 
so as to take advantage of their better qualities while at the same time 
structuring situations so as to minimize their weaker qualities. In this 
case, you can use your experience and your interpersonal skills to make 
allowances for the potential users' range of professional styles and thus 
to Increase the likelihood of evaluation use. 

As always, you may refer to our complete sample worksheet (Figure 14) 
if you wish. 

(Insert Figure 19) 

RECAP ; Let's take stock of where we are In our planning for 
evaluation use. You've now completed the first two of the 
four factor clusters. In Factor Cluster A, you defined the 
background against which the evaluation will be played out; 
In FactorCluster B, you Identified and organized the players. 
Now you know who will be Involved, and what the pertinent 
characteristics and abilities of these Individuals are. You've 
also noted Information about these potential users that should 
be considered as you plan for the actual evaluation activities. 
And, most Important, you've Identified what you need to look for 
In an e valuator to assure that he or she will be use-oriented. 
You should also have a general Idea what direction the planned 
evaluation will take. So, everything Is now set to go — the 
players are In place, and the plan of action Is ready. In the 
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next chapter, we'll get your evaluation rolling. Once It begins, 
you'll have less time to sit back and think about other ways to 
structure the process. So please take some time to make sure 
that you feel okay about the way you've filled out the worksheets 
thus far. Go back and make any changes you want before going 
on to the next chapter. 

If you're Impatient to see how the evaluation Is likely to unfold, then 
go ahead and turn to Chapter IV, Factor Cluster C. If, however, your 
evaluation Is still not off the ground — for Instance, If your evaluator 
has not yet been selected, or If you are only at the beginning stages of 
organizing an evaluation of the project for which you are responsible — 
don't despair of being left out as you turn to Chapter IV. He haven't 
forgotten about you! You'll find specific Information and helpful advice 
geared to your situation throughout the rest of the workbook. 
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Organizing For Evaluation Use 6 

Worksheet B1 



Fiqure 1.3L 



Evaluation Topic: J>ouaJL Akickto 

(JUJlAJUCiliUffKJ 



Factor Cluster B: 



User interest in evaluation 



User commitment to use 



Evaluator characteristics (selected) 



Evaluation procedures— plan 



User professional style(s) 



With respect to my own program and the above topic, I need to keep the following in mind: 



Factor Affecting Use: Relevant Information: 



Things I Can Do To Influence Use: 



B-1. User interest in 
evaluation 

(views about program; 
questions & concerns; 
predisposition to eval./ 
evaluator; need for eval.; 
expectations and risks 
for the eval.) 



2 jKjt(Atht. Jtac^naA^aWi^ai^ 2. 
tut cu>a. AMatt cj Jra euftWwu 



5. 3kt cU|A . dkto*^ uwct AtwUrfCto Jt> 5., 
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Organizing For Evaluation Use 



Figure, 13 



Evaluation Topic: JouaLJkutiu* 



Factor Cluster B: 



User interest in evaluation 



User commitment to use 



Evaluator characteristics (selected) 



Evaluation procedures — plan 



User professional style(s) 



With respect to my own program and the above topic, I need to keep the following in mind: 



Factor Affecting Use: Relevant Information: 



Things I Can Do To Influence Use: 



B-l User interest in 
evaluation 

(views about program; 
questions & concerns; 
predisposition to eval./ 
evaluator; need for eval.; 
expectations and risks 
for the eval.) 



jmunltfi* aAi- ^*\J)ui6tAr^ a>b^ Jo* mjAxxjcju omxtki AudU> 



oIuaW* . lti^*u*ri^ ^a^aw cjuWioia *Cwiaiu> 



2. JKjl 



3. 3a oUjjt. cbc6vnA uu&rit ui> Watt 3. 7)[aJic juj*jl vtfte, eualuaArb 
tut ct6 a Ajte-tii cl fa evakiaivkj ainctwahuulo -flu**, pnoqwunu 
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Organizing For Evaluation Use 

Worksheet B2 



Figure. )4 



Evaluation Topic: JaaaLMudiifi 
tunrucuJUumj 



Factor Cluster B: 



User interest in evaluation 



User commitment to use 



Evaluator characteristics (selected) 



Evaluation procedures — plan 



User professional style(s) 



With respect to my own program and the above topic, I need to keep the following in mind: 



Factor Affecting Use: Relevant Information: 



Things I Can Do To Influence Use: 



B-JL User commitment 
to use 



Skjuuujl jWMJfuurJo Jhe. Job cuncL 
^LdbJhLjwuL Jfo tMfak> JtaaidtkrtAj 

2. ^jrfxAUKtdjjmhvi* 2. 
fyzt Jhdt at iwn*t ailict Jft**u 
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Organizing For Evaluation Use 



Figure. 15 



Evaluation Topic: Jiouai Mudw> 
VUA/wcAdJUxru 



Factor Cluster B: 



User interest in evaluation 



User commitment to use 



Evaluator characteristics (selected) 



Evaluation procedures— plan 



User professional style(s) 



With respect to my own program and the above topic, I need to keep the following in mind: 



Factor Affecting Use: 



B-2. User commitment 
to use 



Relevant Information: 



Things I Can Do To Influence Use: 



Jkcauu, Lju>AiurJo J^^cjuC jMkm^^CL an^aifti^ 
fyjdo The jwuL hjudnirfukj )utaufoi^ 

2. Jh^j^UMicea> teachn* oalJAzaL 2. bpftajv at <up.d*U*^na , 

3 JKjl jniu;jhca£hi06 a^arUKiuumAc 3 j/iu jo <^tt ram. ^ Jta, ^ntur jfeaJtKo 
about ;tfu. ^o^iniJ^ Jhat jtfu. amki*^ u/ttfb ^Wl jvtduak/u 

4. ^^^jiw^uA^i^ 4. TTjaiit^ii^Jtfa evalwhru 
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Organizing For Evaluation Use 



Figure Ik 



Evaluation Topic: Mcud-Mtju)tiu> 



Factor Cluster B: 



User interest in evaluation 



User commitment to use 



Evaluator characteristics (selected) 



Evaluation procedures — plan 



User professional style(s) 



With respect to my own program and the above topic, I need to keep the following in mind: 



Factor Affecting Use: Relevant Information: 



Things I Can Do To Influence Use: 



B-3» Evaluator char- 
acteristics (selected) 

(background/identity; 
credibility; choice of role; 
willingness to involve 
user; commitment to use; 
political sensitivity) 



1. *Wt JtucfcidLcMit - oua> djuJ/uct - 



3. 



4, 
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3 



Figure, 11 



Evaluation Topic: J^ocjuzl^kdao 



Factor Cluster B: 



User interest in evaluation 



User commitment to use 



Evaluator characteristics (selected) 



Evaluation procedures — plan 



User professional style(s) 



With respect to my own program and the above topic, I need to keep the following in mind: 



Factor Affecting Use: Relevant Information: 



Things I Can Do To Influence Use: 



B-3. Evaluator char- 
acteristics (selected) 

(background/identity; 
credibility; choice of role; 
willingness to involve 
user; commitment to use; 
political sensitivity) 



1. WtJbuuckuLotit- owv dufruct - 1. 
Aahc4>afe jr\o ^aoWuao wit. 



2. 



3. 



4. 



5. 



r 
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Organizing For Evaluation Use 




Evaluation Topic: j^oou^Mudu* 

CMA/UUjlw*i> 



Factor Cluster B: 



User interest in evaluation 



User commitment to use 



Evaluator characteristics (selected) 



Evaluation procedures— plan 



User professional style(s) 



With respect to my own program and the above topic, I need to keep the following in mind: 



Factor Affecting Use: 



EM. Evaluation 
procedures— plan 



Relevant Information: 



Things I Can Do To Influence Use: 



1. %<w ta*o Juot jrnjuu^njL %l> 
♦ SAT Mtyub ? 

2. JJtwJti MX, JJ*b ^ -fikuUni 



3. Wur uihtit At axAAicuitiwu cWbiCL 3. 



<5~l 
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Organizing For Evaluation Use 

Wofeteet B5 



Evaluation Topic: Joaal Jhdiio 



Factor Cluster B: 



User interest in evaluation 



User commitment to use 



Evaluator characteristics (selected) 



Evaluation procedures— plan 



User professional style(s) 



With respect to my own program and the above topic, I need to keep the following in mind: 



Factor Affecting Use: 



B-5. User professional 
style(s) 

(administrative & organi- 
zational skills; initiative; 
openness to new ideas) 



Relevant Information: 



Things i Can Do To Influence Use: 



1- ^vjl dU^t. ctai/L> Juu> h jjt ^yiodfaA 1. 
Jo aft jnvtwfi } but or^jjfomXxMtSA 

2. %x fyoofidtM ttmrm)^ - Curui 2. 

3. %l Aukufo acuat (x^jtm^o chouru^L 3. 



5. 



Jo 3mi\AwtMn 

Vu^tur" onto J&uhjxto jmakz* 
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IV. OPERATIONALIZING THE INTERACTIVE PROCESS: 
FACTOR CLUSTER C 
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Organizing For Evaluation Use 

Woteteet CI 



Evaluation Topic: 



Factor Cluster C: 



Evaluation procedures — execution 



Substance of evaluation information 



Evaluator commitment to use 



Information dialogue— formative 



User information processing preferences 



With respect to my own program and the above topic, I need to keep the following in mind: 



Factor Affecting Use: Relevant Information: 



Ol Evaluation 
procedures— execution 

(methods to be used; 
dealing with mandated 
tasks) 



Things I Can Do To influence Use: 



3, 
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Organizing For Evaluation Use 

Worksheet C2 



Evaluation Topic: 



Factor Cluster C: 



Evaluation procedures — execution 



Substance of evaluation information 



Evaluator commitment to use 



Information dialogue— formative 



User information processing preferences 



With respect to my own program and the above topic, I need to keep the following in mind: 



Factor Affecting Use: Relevant Information: 



C-2. Substance of 
evaluation information 

(specific information 
selected; information 
relevance to audience) 



Things I Can Do To Influence Use: 
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Organizing For Evaluation Use 

Worksheet C3 



Evaluation Topic: 



Factor Cluster C: 



Evaluation procedures — execution 



Substance of evaluation information 



Evaluator commitment to use 



Information dialogue— formative 



User information processing preferences 



With respect to my own program and the above topic, I need to keep the following in mind: 



Factor Affecting Use: Relevant Information: 



C-3. Evaluator 
commitment to use 



Things I Can Do To Influence Use: 



1. 



2. 



3. 



4. 



5. 



5. 
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Organizing For Evaluation Use 

Wosrksihse'l C4 




Factor Cluster C: 



Evaluation procedures— execution 



Substance of evaluation information 



Evaluator commitment to use 



Information dialogue— formative 



User information processing preferences 



With respect to my own program and the above topic, I need to k eep the following in mind 
Factor Affecting Use: Relevant Information: Thinge I Can Do To Influence Uee: 



C-4. Information 
dialogue— formative 

(amount & quality of inter- 
action between evaluator 
and user(s)) 



1. 



2. 



3. 



5. 
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Organizing For Evaluation Use 

Worksheet C5 



Evaluation Topic: 



Factor Cluster C: 



Evaluation procedures—execution 



Substance of evaluation information 



Evaluator commitment to use 



Information dialogue — formative 



User information processing preferences 



With respect to my own program and the above topic, I need to keep the following in mind: 



Factor Affecting Use: Relevant Information : 



C*5. User information 1. 
processing preferences 



2. 



Things I Can Do To Influence Use: 



2. 



3. 



3. 



4. 



5. 
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CHAPTER IV 

OPERATIONALIZING THE INTERACTIVE PROCESS: - FACTOR CLUSTER C 

We've come to a transition point 1n our organizing for evaluation use. 
Chapters II and Il^on Factor Cluster. A and B, refer mainly to the planning 
stage of the evaluatlon^to those activities that are carried out prior to ■ 
the actual conduct of the evaluation. This chapter and the next deal 
mainly with conducting and reporting the evaluation Itself. Now you must 
begin to consider the period when the evaluation processes actually take 
place. Prior to this point you were anticipating future actions; the time 
for carrying out these actions 1s now upon you. Here 1s where you'll put 
Into practice the steps you've planned, guided by the discussion of Factor 
Cluster A and B. If you've done your planning job well, then the evaluation 
should proceed smoothly; 1f not, It's still not too late to make some mid- 
course adjustments. 

The activities suggested as you master the content of Factor Cluster C 
will, to a large extent, determine the evaluation Information available for 
use. Thus, the questions of what that Information actually comprises and 
of how 1t gets transmitted to potential users become Increasingly Important 
to the Issue of evaluation use. 

Now that you've had a brief preview of things to come, let's examine 
Factor Cluster C Itself, reproduced below: 
(Insert Factor Cluster C) 

You will note that 1t 1s made up of five Individual factors, each 
separated for our consideration by a solid line. As we've said before, 
though treated separately for the sake of convenience, the factors 1n each 
pattern are highly Interactive^ and this 1s particularly true of Factor Cluster 

c. 
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Thus, 1n thinking about your projects evaluation, you may find it somewhat 
difficult to isolate each of these factors. The discussion that follows will 
point out some of these areas of interrelationship. 

(Insert Diagram 10) 

The first factor, termed "Execution of evaluation procedures," can be 
regarded as the organizing element for Factor Cluster C, in the sense that 
all other activities and interactions necessarily and logically flow from the 
carrying out of the evaluations processes. To focus your thinking so that 
you can fill out Worksheet CI, however, we will ask you to view this factor 
as involving a set of fairly mechanical tasks: for example, we would include 
in this element such tasks as establishing a data collection schedule and 
adhering to it; taking steps to satisfy the information requirements of 
external agencies, setting specific meetings or other events at which the 
evaluator must be present, and the like. Each evaluation entails its own 
particular set of such tasks, and no two evaluations will be alike in this 
respect. 

We ask you now to think of your own projects evaluation plan and to 
identify, 1n the second column of Worksheet CI, that set of tasks required 
by the evaluation. A further word is perhaps necessary before you begin to 
write: 1t may be that the evaluation of your project 1s already 1n progress 
and thus you will have no particular difficulty 1n filling out Worksheet CI. 
If so, then go ahead and begin this task now. If, on the other hand, you have 
had little experience with evaluations or 1f your projects evaluation has not 
yet begun, then you might want to read the next two paragraphs before filling 
out the worksheet. 

Those of you who are Inexperienced 1n evaluation or whose evaluation 1s 
not yet under way may want to deal with Worksheet CI (at the beginning of this 
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chapter) in a slightly different way than that described above. It would 
probably be more helpful if you used this worksheet to identify some key 
tasks that should be taken care of at some later, more appropriate, time. 
Use the space on the worksheet to "flag" these matters for your future 
attention. In your case, it would be all right »ot to have a great deal of 
information recorded on Worksheet CI. . (By way of illustration, look at the 
way we've filled out Figure 20.) 

For example, let's suppose that you have administrative responsibility 
for a new project and that the evaluator has yet to behired. At this point, 
you could use Worksheet CI to. note any evaluation procedures already deter- 
mined by prior evaluation bounds or constraints (e.g., if a specific measure 
is to be administered, what arrangements must be made? If particular groups 
or individuals are to be surveyed or interviewed, what schedule should be 
followed?) or any questions you may have about evaluation procedures (e.g., 
what are the pitfalls to be avoided in carrying out a survey? Are there any 
special arrangements which must be made in order to administer a particular 
measure or carry out other data-gathering activities?). These notes will 
serve as reminders to you later on. 

(Insert Figure 20) 

Please observe that in the Appendix to this workbook, we have provided 
a second set of blank worksheets for this chapter. The first of these is 
labeled "Worksheet CIA". When the time is right— that is, when the evaluation 
of your project has progressed to this point— you should go back to the work- 
sheet and fill it out completely. 

If you have not already done so, please complete Worksheet CI now before 
going on to the next section.- 
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(Insert Diagram 11) 

The second element of Factor Cluster C is termed. "Substance of evalua- 
tion information." As your evaluation procedures are implemented, evaluation 
information will be produced. Most of this information should be on target; 
i.e., information pertaining to the evaluation topic(s) which you and your 
users have previously identified. Occasionally, however, an evaluation 
procedure will produce information not related to the questions under consider- 
ation. You will then have to consult with both the evaluator and the users to 
determine whether this information does indeed have some relevance or interest, 
or whether it should be omitted from the totality of information to be 
communicated as part of the evaluation findings. 

Another aspect of this factor concerns the deliberate targeting of . 
information toward specified users. Not all potential users are equally 
interested in all the evaluation topics. Part of the task in organizing for 
use is to match as closely as possible information to user. Your evaluator 
may need your expert guidance in this area. 

Let's look at an example of how this works. Imagine that you are the 
commanding officer of an infantry battalion which is providing training for 
long-range reconnaissance patrols. You have identified your potential users 
as the superior officers, the officers who command the training companies, 
the non-commissioned officers (NCOs) who lead the training patrols, and the 
soldiers in these patrols. The superior officers will be interested in infor- 
mation pertaining to the full range of evaluation concerns (e.g., the effective 
ness of the training provided to enlisted personnel, their performance in the 
field, the performance of their superior officers and NCOs.) The company 
commanders will be interested in how their companies stack up against other 
companies; the NCOs will- be interested in how their units compare with other 
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units 1n the company; and the soldiers themselves will be Interested mainly 
1n their own individual performances. In these circumstances, your evaluator 
needs to know which group should get which evaluation data: that 1s, to 
target specific Information to the specific audience interested 1n that 
Information. Information which 1s not pertinent to a particular user. 1s 
likely to go unused — a situation you should be doing your utmost to avoid. 
Moreover, 1f the relevant information 1s burled In a mass of irrelevant (to a 
particular user) Information, 1t may be overlooked and thus will stand no 
chance of being translated Into action. The Individual user's attention must • 
be carefully focused on, or directed to, the Information that 1s relevant to 
him/her. 

Now, consider your own project, and the users you have Identified (not 
forgetting, of course, yourself). What should be the focus of Information to 
be received by each of these parties? What information should be emphasized 
for a particular user? What information is not likely to be relevant to 
answering a particular user's questions or concerns about the project? 

When you have sufficiently analyzed the needs of your users for targeted 
information, turn to Worksheet C2 and fill out the second column, "RELEVANT 
INFORMATION." If, again, your evaluation is not yet 1n progress, you may 
choose to fill out the worksheet in terms of what you know now and to defer 
filling out the complete worksheet (labeled C2A in the Appendix) until a 
later time, when your evaluator is just about to enter the main phase of the 
evaluation. We have filled out a sample form in Figure 21. 

(Insert Figure 21) 

As you turn your attention to the third column of Worksheet C2, "THINGS 
I CAN DO TO INFLUENCE USE," you may feel that any actions you can take in this 
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sphere are limited. Generally speaking, the evaluator should take charge of 
this aspect of the evaluation. He/she 1s assumed to have a certain expertise 
in these matters, and you certainty shouldn't have to do both your own job 
and the evaluator's 1f you can avoid 1t. Somet1mes r hov^ever, due to 
circumstances beyond your control, the evaluator may not be able to perform 
this function satisfactorily. In such cases, 1t w1H be up to you to make 
sure that the right Information gets to the right user. If you do not know 
yet who your evaluator .is to be or just how the evaluation is to be conducted, 
you may not be able to fill in this column now. As we have said before, 1t is 
all right to leave blank columns where you lack sufficient information. 
Remember that the object of this exercise is to help you 1n your thinking about 
and planning for evaluation use. And we want you to fill out these worksheets 
with your own. project evaluation in mind. 

(Insert Diagram 12) 

As we Indicated above, the evaluator's attitude ^toward .obtaining and 
communicating appropriate information to users is- important in promoting 
evaluation use. Thus, the next factor in Factor Cluster C is, again, the 
evaluator's commitment to use, which was initially disc.ussed as part of 
Factor Cluster B (refer to p. to freshen your memory). We bring 1t up 
again here to emphasize how crucial this factor is to evaluation use — and 
also to demonstrate how vital it is in operatlonalizing the interactive portion 
of the evaluation process. If your evaluator is committed to evaluation use, 
he or she will need only minimal assistance in selecting on-target information 
and communicating that information to users. If, however, the evaluator is 

♦ 

less than deeply 'committed to this goal, you may have to push from behind. 
At this point, you should be taking stock of how things are going (or how 
things look like they will be going), based on your present knowledge of the 
$ evaluator's operating style. 
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This factor 1n Factor CI uster C allows for mid-course correction, if 
needed. You may have had high hopes for your evaluator when he/she was first 
hired. Have these first impressions panned out? If not, what can you do to 
facilitate the process of information handling? When you feel you have 
sufficiently considered your evaluator's attitude toward use and current " 
performance on the job, then fill out the second column of Worksheet C3, as 
we have done in Figure 22. If, however, your evaluation is not yet under way, 
then you may want to wait until later to fill out this worksheet. If you do • 
decide to wait, don't forget you will need to return to this point after the 
evaluation gets underway. You've already invested considerable time and 
energy in this workbook, and you shouldn't settle for anything less than the 
maximum possible benefit you can get from it. 

(Insert Figure 22) 

Now, if you've gone ahead and filled out part of Worksheet C3, ask 
yourself if there is anything you either need or want to do do further the 
evaluation process or increase the likelihood of evaluation use. Even in the 
happy event that your evaluator is. functioning well, maybe you see something 
that could be done to make things go even better, something which did not 
occur to you earlier when you filled out the Factor Cl uster. B worksheets. If 
so, write that information now in the third column of Worksheet C3, "THINGS I 
CAN DO TO IMPROVE USE." 

(Insert Diagram 13) 

At tMs point in your thinking, you are probably beginning to see the 
interrelated nature of the components of Factor Cl uster C. The evaluator's 
approach to carrying out the evaluation tasks, the type andanount of information 
he/she selects and targets to particular users, and his/her attitude toward 
evaluation use are closely interdependent. In fact, these factors, in combina- 
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tion, often determine the fourth factor in this cluster, labeled "Formative 
information dialogue." Let's clarify what we mean by these terms. 

"Formative" evaluation usually takes place in the early years of a 
project's operation, when it is still in a developmental stage (i.e., before 
it has become "fixed" and no further modifications are possible or permitted) 
and is intended to provide information which can lead to project improvements. 
It assumes that decisionmakers will use- the evaluation information when 
considering what changes, if any, to make in the project being evaluated. • 
(The other major category of project evaluation is called "summative evaluation", 
and will be discussed in Chapter V). 

The term "information dialogue" refers to the give-and-take between • 
the evaluator, yourself, and other potential users during the early, develop- 
mental phases of project operation. Such exchanges can be formal (as in a 
scheduled meeting) or informal (as in a brief hallway encounter) and can take 
a broad range of forms: written reports, telephone calls, requests for 
clarification, the sharing of a draft before it is finalized, a discussion of 
evaluation progress over lunch. These kinds of interactions, whether formal 
or information, can be initiated by the users as well a? by the evaluator. 

Such dialogue can vary in both quantity (the number of such interactions, 
their length, etc.) and quality (the appropriateness of the information, the 
interpersonal communication skills of the participants, etc.) Both the 
substance of the information and the nature of the interaction should be of 
appropriate quality. For example, a high-quality interation might be a . 
discussion between the evaluator and yourself which explores how the evaluation 
is going and what its ultimate findings might look like. Or the evaluator might 
spend some time with the project, administrator- analyzing interim findings. 
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Interpreting their meaning, and discussing possible recommendations for the 
final report. 

Only you can decide what constitutes the best mode of dialogue and 
Interaction with respect to your project, your evaluator, yourself, and other 
potential users. You should take some time to discuss this Issue with your 
evaluator as soon as the evaluation plan 1s operational 1 zed, 1f you haven't 
already done so before that time. Remember, too, that the goal of ^ formative 
Information dialogue 1s two-fold: first, to provide Information which will 
be relevant to users' concerns and questions about the program at a stage 
where changes can still be made; and second, to reinforce the developing 

* 

relationship between a caring, committed evaluator and a caring and committed 
set of evaluation users. 

With these goals 1n mind, turn now to Worksheet C4. . If your evaluator 
Is already hired and the evaluation Is 1n progress, you should have no difficulty 
filling out the "RELEVANT INFORMATION" column now. But again, 1f your evaluator 
has not started work, then you might want to use this worksheet Instead to jot 
down your Initial thoughts about the types of Interactions most suitable for 
Imparting evaluation Information to your potential users. In this event, don't 
forget later to go back to the Appendix and fill out Worksheet C4A as the 
actual evaluation process unfolds. The completed sample worksheet 1s 1n Figure 
23. 

(Insert Figure 23) 

After you have filled out the second column, consider whether you need 
to Intervene 1n the types of Interactions that have been occurring. Perhaps 
only Informal exchanges have taken place to this point, and you see a need for 
some additional structure. Perhaps there have been structured sessions but 
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they have not addressed the more significant evaluation questions. Perhaps 
the users need more exposure to the evaluator; you may want to arrange a 
working lunch or some other Informal event. Perhaps you want to Impose more 
structure on the dialogue by requiring that any written materials go through 
your office prior to release. Obviously, no two evaluation situations will 
entail the same formative evaluation dialogue,- and we do not want to limit 
your Ideas on this point. Once you've reviewed your own needs and desires, 
continue on to fill out the third column In Worksheet .C4, "THINGS I CAN DO 
TO INFLUENCE USE." If your evaluation Is not yet under way, use this column 
to list any particular strictures on dialogue and communication that you wish 
to Impose, given your particular constellation of program factors. 

(Insert Diagram 14) 

Next we turn our attention to the final factor In Factor Cluster C,. 

» 

"User Information processing preferences. V. we have already; indicated, .Infor- 
mation should be targeted to Individual users or groups of users. Here, we 
are suggesting that the Information should be presented In a format which Is 
acceptable to those users. What do we mean by an acceptable format? Each of 
us, as individuals, has a preferred way of dealing with and processing Incoming 
information. Some of us are visually oriented, others are auditorily oriented, 
and still others respond best to a combination of both visual and auditory 
stimuli. Some of us comprehend numbers more quickly than words, though for 
most people the opposite Is true. In addition, people have preferences for 
different styles of Interaction. Some are more formal and rank-conscious, 
especially within an organizational setting, while others do not stand on 
ceremony regardless of the context of the Information '. presentation. When 
planning and carrying out the evaluation's Interactive process, you and your 

» 

evaluator need to be conscious of these Individual preferences. 
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For example, suppose that you are operating within an organization 
which 1s rather rank-conscious. In this case, your users will probably prefer 
a formal report, delivered by the e valuator him/herself, over an Informal 
report, delivered In a hallway by a junior participant In the evaluation process 
Similarly, If "ome sort of board or committee Is to be Involved In a formal 
reporting situation, your purposes will probably best be served If you adopt 
a variety of techniques: for Instance, an oral presentation, accompanied by 
visual aids such as charts and graphs. The point 1s that there should be a 
match between users' preferences for a given format or mode of presentation, 
and the way In which evaluation Information Is "packaged." The results of 
a basically sound and use-oriented evaluation can wind up being discredited - 
or disregarded If the findings are not presented In a manner consonant with 
user preferences as to Information-processing preferences. 

Take a moment now to reflect upon what you know about the Inform*- ' 
tton processing preferences of the various potential users and groups of 
users (don't forget to Include yourself). Then go on to fill out Worksheet 
C5. As you are filling out column 3, "THINGS I CAN 00 TO INFLUENCE USE," 
note that while you can probably do nothing to change a given person's 
preferred Information-processing style, you can certainly take steps to see 
that the format In which the evaluation Information 1s presented to that 
person 1s appropriate to that style. You can probably complete Worksheet C5 - 
even If your evaluation Is not yet under way, since you have already Identified 
the potential users back on Worksheet A2. You'll find our completed sample 
worksheet 1n Figure 24. 

(Insert Figure 24) 
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RECAP ; Hell, by now you are probably In the thick of things. The 
evaluation Is under way. Information Is being collected, processed, 
and communicated to specific users or user groups, we have labeled 
Factor Cluster C an "Interactive process" for good reason. Each of 
the factors we've Identified and discussed Is closely related to all 
the others. After all, you can't carry out evaluation procedures 
without generating Information; you can't collection Information without 
specific content; you can't target this Information without a 
knowledge of the users who need or want It; and you can't present 
this Information effectively without knowing how the users prefer 
to receive It. Finally, you can't guide the overall process of 
Information dialogue and exchange without knowing how the partici- 
pants feel about the forms and modes this Interaction should take. 

You may find that your evaluation remains at the stage outlined In 
Factor Cluster C for some time. Not all evaluations are accomplished within 
a brief time span. If that's the case, you may want to go back periodically 
and revise the worksheets you've filled out for this chapter of the workbook. 
Your thoughts on evaluation processes may change substantially as the evalua- 
tion evolves. That's .why we've provided a duplicate set of worksheets In the 
Appendix. 

Factor Cluster. C Is marked with a right arrow at Its conclusion, pointing 
us on to the final factor cluster. When you are ready, turn to Chapter V and 
Factor Cluster D. 
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Organizing For Evaluation Use 
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Figure. 3lo 



Evaluation Topic: Jvcurt Muduio 



Factor Cluster C: 



Evaluation procedures— execution 



Substance of evaluation information 



Evaluator commitment to use 



Information dialogue— formative 



User information processing preferences 



With respect to my own program and the above topic, I need to keep the following in mind: 



Factor Affecting Use: Relevant information : 



Things I Can Do To Influence Use: 



Ot Evaluation 
procedures— execution 

(methods to be used; 
dealing with mandated 
tasks) 



1. fcvaWta 1. 

2. tvalu^r^jfu^ hdo toot a^W)2. 



3. Ito^jUfc^ 3. 



5. 
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Organizing For Evaluation Use 

Worksheet C 



figure, 2.1 



Evaluation Topic: JkucAMudxjbo 



Factor Cluster C: 



Evaluation procedures — execution 



Substance of evaluation information 



Evaluator commitment to use 



Information dialogue — formative 



User information processing preferences 



With respect to my own program and the above topic, I need to keep the following in mind: 



Factor Affecting Use: 



C-2. Substance of 
evaluation information 

(specific information 
selected; information 
relevance to audience) 



Relevant Information: 



Things I Can Do To Influence Use: 
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Organizing For Evaluation Use 

Worksheet C3 



Figure 41 



Evaluation Topic: JocudMuduo 



Factor Cluster C: 



Evaluation procedures — execution 



Substance of evaluation information 



Evaluator commitment to use 



Information dialogue— formative 



User information processing preferences 



With respect to my own program and the above topic, I nee d to keep the following in mind: 
Factor Affecting Use: Relevant Information: Things I Can Do To Influence Use: 



C*3. Evaluator 
commitment to use 
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Organizing For Evaluation Use 

Wosfefoae! C4 



Figure 2.3 



Evaluation Topic: JoutdJhdm 



Factor Cluster C: 



Evaluation procedures— execution 



Substance of evaluation information 



Evaluator commitment to use 



Information dialogue— formative 



User information processing preferences 



With respect to my own progr am and the above topic, I need to keep the following in mind 
Factor Affecting Use: Relevant Information: Things I Can Do To Influence Use: 



C-4. Information 
dialogue— formative 

(amount & quality of inter- 
action between evaluator 
and user(s)) 
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Organizing For Evaluation Use 

WosteteeS C5 
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Evaluation Topic: JoaaJiMudm 



Factor Cluster C: 



Evaluation procedures— execution 



Substance of evaluation information 



Evaluator commitment to use 



Information dialogue— formative 



User information processing preferences 



With respect to my own program and the above topic, I need to keep the following in mind: 



Factor Affecting Use: 



Relevant Information: 



Flings I Can Do To Influence Use: 
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V. ADDING THE FINISHING TOUCHES 
FACTOR CLUSTER D 
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Organizing For Evaluation Use 

Worksheet D1 



Evaluation Topic: 



Factor Cluster D: 



Evaluation reporting 



Evaluator characteristics (selected) 



Information dialogue— summative 
User commitment to use 



With respect to my own program and the above topic, I need to keep the following in mind: 



Factor Affecting Use: Relevant Information: 



D-1. Evaluation 
reporting 

(frequency; timing; format 
of report; statistical /nar- 
rative data; format of 
presentation) 
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Things I Can Do To Influence Use: 



3. 



4. 
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Organizing For Evaluation Use 

Wosteteet D2 



Evaluation Topic: 



Factor Cluster D: 



Evaluation reporting 



Evaluator characteristics (selected) 
Information dialogue — summative 
User commitment to use 



With respect to my own program and the above topic, I need to keep the following in mind: 



Factor Affecting Use: Relevant Information: 



D-2. Evaluator char* 
acteristics (selected) 

(commitment to use; 
political sensitivity) 



Things I Can Do To Influence Use: 



3. 
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Organizing For Evaluation Use 

WoAshset D3 



Evaluation Topic: 



Factor Cluster D: 



Evaluation reporting 



Evaluator characteristics (selected) 
Information dialogue— summative 
User commitment to use 



With respect to my own program and the above topi c, 1 need to keep the following in mind 
Factor Affecting Use: Relevant Information: Things I Can Do To Influence Use: 



D-3. Information 
dialogue— summati* -e 
(amount, type and quality 
of interaction between 
evaluator and user(s)) 



1. 



3. 



4. 
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ieet D4 



Evaluation Topic: 



Factor Cluster D: 



Evaluation reporting 



Evaluator characteristics (selected) 
Information dialogue— summative 
User commitment to use 



With respect to my own program and the above topic, I need to keep the following in mind: 



Factor Affecting Use: Relevant Information: 



D-4. Uaer commitment 1 . 
touae 



Things I Can Do To Influence Uae: 



1. 



2. 



5. 
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CHAPTER V 

ADOING THE FINISHING TOUCHES: FACTOR CLUSTER 0 

Factor Cluster D, the last of the factor clusters, represents the 
final phase In maximizing the potential for evaluation use. This factor 
duster belongs to that point In the evaluation process where most (If not 
all) of the Information has been collected. In the earlier phases, you 
considered those characteristics of the system and of users that might 
make a difference In evaluation Impact. He trust that your evaluation has 
benefited from the steps you have taken on the basis of these considerations 
and that your project has benefited from the evaluation process Itself and 
from any mid-course corrections made on the basis of Informal formative 
evaluation data. Now, In Factor Cluster D, the fruits of the evaluation 
process are displayed for all to see and to comment upon. 

Of course, there Is still work to be done. Once all the Information Is 
available, It must still be cawunlceted effectively to the right people, 
presumably the Identified users. The users must live up to their designation 
and actually use the Information In some meaningful way. As you might expect 
from this brief rationale, the components of Factor Cluster D are more 
heavily weighted toward human factors once again. After all, people are the 
users of any evaluation. They determine for themselves whether It will be 
put to action use, conceptual use, or both (to recall our discussion In 
Chapter I). Of course, you will have done your best throughout the course of 
the evaluation to Influence the outcome In favor of the use which you feel 
to be most appropriate. But we are getting a bit ahead of ourselves. Let's 
go back and examine the factors that make up this final cluster. 

(Insert Factor Cluster D) . f 
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Each of the four factors In Factor Cluster 0 Is Important to making 
use happen. He will consider each of them In turn. 

(Insert Diagram 15) 

The first' of the four factors, "Evaluation reporting," refers to any and 
all methods of communicating findings to users: a written report, an oral 
presentation, an Informal staff session with the evaluator, and any other 
form of Interchange whose objective Is to communicate evaluation findings. 
Any formal constraints which you or some agency may have previously Imposed 
on the evaluator should also be considered here. Think back to our earlier 
example of the bilingual project director (Chapter II). This administrator 
specified In the evaluator's contract that a special report directed to the 
School Board was to be produced, regardless of any other method used to 
communicate evaluation findings. This Is one example of a formal constraint 
on the evaluation reporting process. 

The term as used here, however, refers mainly to the end-of-e valuation 
reporting situation. In our earlier discussion of the ongoing dialogue 
between evaluator and users, we refrained from using the term "reporting" 
precisely because we wanted to reserve the term for the final factor pattern. 
The elements to be considered In connection with this factor are: the 
frequency of reporting, the timing of the report(s), the foimat of the 
report(s), and the format of the reporting presentations ). 

Let's define these terms a bit more narrowly. He have already come out 
In favor of ongoing dialogue between evaluator and users as a major deter* 
mlnant of evaluation use. Similarly, the frequency with which evaluation 
findings are communicated contributes to the credibility of the evaluator, 
the credibility of the findings, and the potential for obtaining Information 
which Is responsive to users' needs. 
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The tiding of an evaluation report, be It In written or oral form, can 
also be crucial to evaluation use. Obviously, a report which does not cone 
out until after a project decision deadline has passed cannot contribute to 
that decision. Nor can a report which comes out before a deadline but which 
does not contain Information relevant to decisionmakers be regarded as useful, 
well-timed reports during the course of an evaluation can, however, be useful 
In two ways: they can provide partial answers to user questions, and they 
can arouse user anticipation for additional evaluation Information. Obviously, 
then. Interim reports go a long way toward fostering evaluation use and 
enhancing the evaluator's credibility en route to the final report. 

The format of a report can also affect Its credibility, and hence Its 
potential for use. Here, we are referring mainly to written reports. The 
organization of the report, the balance between statistical data and narra- 
tive, the use of visuals or other such devices, and the style (Including use 
of jargon) can affect a reader's perception of the Information contained In a 
report. Cramon sense dictates a match between the format of a written report 
and the preferred Information processing style of the Intended recipients. 
I.e., the users. 

The final component of the evaluation reporting factor Is the format of 
any oral presentation of the findings. The mode of presentation (e.g., should 
the presenter read his speech from a manuscript or speak extemporaneously 
from notes?), the use of visual aids (e.g., should the speaker use graphs 
and charts?) should these be passed out to the audience, or should overhead 
projections be used?), and the size aad Involvement of the audience (e.g., 
should the speaker take questions from the floor? during the presentation 
or only at the end?) may all affect the audience's reaction to, and potential 
use of, the Information presented. 
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Now that you have some Idea of the Importance of evaluation reporting, 
consider where your own project evaluation stands. Has there been a satis- 
factory match In the style of previous Interchanges between the evaluator and 
yourself, as well as with the other users? Have you received any Interim 
written reports from the evaluator? If so, how do they strike you? Has the 
evaluator made any oral presentations to a user audience? If so, how did 
they go over? Are there any special reporting needs to which the evaluator 
should pay special attention? Do you need to Intervene In the process? 

When you have sufficiently analyzed your present evaluation situation, 
go on to fill out worksheet 01. In the event that your project evaluation 
Is not yet under way or has net yet reached this stage, use this worksheet 
to note any relevant Information In terms of reporting needs. Don't forget 
to go back later and redo worksheet DIA when your project evaluation has 
progressed to that point. 

If you need to, refer to cir completed sample worksheet In Figure 25 below: 
(Insert Figure 25) 

Why not fill out Worksheet Dl now? 

(Insert Diagram 16) 

You'll remember that, because of their Importance, some factors are 
Included In more than one factor cluster. Such Is the case with the second 
component of Factor Cluster D, which was first discussed as part of Factor 
Cluster B: the evaluator's commitment to use and political sensitivity. 
(Both were considered under the general rubric of "Evaluator characteristics"). 
Let's take a moment to refresh your memory about these elements. 
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Your evaluator must have a sincere commitment to use if the whole 
process is to function optimally. Failing such a commitment from your 
evaluator, it behooves you to take an extremely active role in promoting 
evaluation use. Now that the final stage of the evaluation process has 
been reached, efforts to encourage potential users to avail themselves of 
the evaluation information — produced at such expenditure of time and 
energy — cannot slacken. 

In addition, the evaluator must understand the political environment 
in which the evaluation is taking place so that the recommendations in 
the final report are feasible, practical, and consonant with the goals and 
resources of your organization. Common sense should tell you that recommen- 
dations that do. not lie within the capabilities of your organization do not 
stand much change of being implemented. Moreover, such recommendations 
reflect negatively on the evaluator's credibility, and, by extension, on 
the credibility of the evaluation findings themselves (and perhaps on your 
credibility as well). Clearly this situation should be avoided, and the 
way to avoid it is by doing precisely what we hope you have been doing all 
along - taking the time, the care, and the energy to supervise and shepherd 
the evaluation right from its inception. 

Reflect back now on your evaluator's commitment to use and on his/her 
political sensitivity. What are the prospects for having suitable recommen- 
dations developed for the final report? If you foresee difficulties, do you 
need to intervene to straighten things out? Go on now and list your thoughts 
on Worksheet Dl, as we have done below. 

(Insert Figure 26) 
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Again, If your project evaluation Is not yet under way, or has not yet 
reached this point In Its progress, use this worksheet to note any concerns 
you may have regarding this factor. We've provided Worksheet D2A In the 
Appendix for your subsequent use at the appropriate later time. 

(Insert Diagram u) 

The third factor 1n Factor Cluster D 1s, once again, a repeat of a pre- 
viously discussed factor, but with a new twist: "Summatlve Information 
dialogue." Earlier, 1n Chapter IV, we talked about "Formative Information 
dialogue" as a component of Factor Cluster C, defining a formative evaluation 
as one which takes place when a project 1s still developing and which Is 
Intended to produce Information that will be used for project Improvements. 
A "formative Information dialogue," then, Is the Interchange (between 
evaluators and users, Including yourself) that takes place at that stage of 
an evaluation when things are still fairly fluid: T.e., the evaluation Is 
going on, and changes can still be made In the project and Its operations. 

In the professional literature, the term "summatlve evaluation" Is 
usually applied to an evaluation that takes place when a project Is drawing 
to a close, or at least when 1t has fully stabilized. For purposes of this 
discussion, we are using the term 1n a slightly different sense. We do not 
believe that projects ever become completely stabilized. Nor do we want to 
Imply that projects must forever remain exactly as they were at the time 
of a summatlve evaluation report.. Indeed, such situations are antithetical 
to evaluation use, which Implies that projects will continue to benefit from 
evaluation Information. 

As we view It, then, a summatlve evaluation dialogue commonly takes place 
at a specified time, such as once a year or every two years after the beginning 
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of a project Its major purpose Is not so much project Improvement (though 
that Indeed may be one result of a summatlve evaluation report) as It Is a v 
summary and review of the project's major features and accomplishments during 
the t1mt perld covered. It focuses on the broad picture rather than on 
details, on aggregate data from a multiplicity of sources rather than on data 
about an Individual project element which may need modification or Improvement. 
(By way of contrast, formative evaluation dialogue tends to focus on details, 
on data about specific project elements.) 

Just as we now recognize the Importance of the formative Information 
dialogue, so too we need to recognize the Importance of the summatlve Infor- 
mation dialogue. It represents the summit of the evaluation process: 
procedures have been planned, data have been collected and analyzed, and 
the Information 1s ready to be communicated directly to those who need or 
want 1t the most. Making sure that such a dialogue occurs between the evalu- 
ator, yourself as responsible administrator, and those users Is a crucial 
phase of evaluation use. 

You can probably think back to some report or other which arrived on 
your desk, sat for some time, and was finally filed somewhere. (Now where 
did you put 1t? On your bookshelf? In the desk drawer? Or...?) Surely, 
you don't want this to happen to your evaluation! You have already taken 
one step toward preventing this unfortunate waste by checking on the match 

« 

between the style and format of the evaluation report and the preferred 
Information processing styles of the users. Another step you can take to avoid 
a situation where the evaluation falls to gain a serious hearing and where 
the report 1s filed away or hopelessly lost 1s to ensure that an on-going 
dialogue between evaluator and users occurs both during the evaluation process 
and at Its natural conclusion. 
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You may find that you want to take an even more active role with respect 
to this factor, particularly If your organization Is fairly formal and conven- 
tlonal. Since you are the responsible administrator (even though your evalua- 
tor may have done most of the evaluation work), you may want to help the 
evaluator select the timing and the method of presenting the findings (not 
select what findings will be presented) for the summatlve (or should we say 
"summation") evaluation report. Only you can decide whether such an action 
1s necessary at this point. 

We think that you are now ready to fill out Worksheet 03. We've gone 
ahead and completed the sample worksheet for you (Figure 27). 

(Insert Figure 27) 

If your evaluation 1s currently 1n progress, then you may or may not be 
ready to deal with this final stage In- the evaluation process. If not, then 
join those readers whose evaluations have not yet begun or have not progressed 
to this point and use this worksheet to note any concerns you may have as to 
how the sumnatlve evaluation dialogue can most profitably take place within 
the context of your project. We have provided an additional worksheet, labeled 
D3A, for your use when the time becomes appropriate. 

(Insert Diagram 18) 

At last! We've come to the last factor In the last factor cluster , and 
again, It's one with which you are familiar: "User commitment to use," which 
was first discussed as a component of Factor duster B. Just think of where 
you are now, and how far we've come: Your evaluation has been planned, carried 
out, and reported directly to those whom you've Identified as users of the 
evaluation. So what's left? Why, the ultimate criterion: What actually happens 
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to this Information, to the evaluation findings? The Ideal situation — the 
one which we've been aiming for — Is to have this Information heard and consi- 
dered seriously by the Intended users. (This does not mean that the evaluator's 
recommendations must necessarily be agreed with or otherwise Implemented.) 
Unless these users are committed to this course of action, then evaluation 
"use" will probably not take place. Again, take a moment to review how the 
potential users stand on this Issue. Do you still need to do some convince? 
Or, because of your careful attention to earlier stages of the evaluation 
process, are they eagerly awaiting their next contact with the evaluator? Do 
they now believe strongly enough In the evaluation that they are willing to 
consider Its findings carefully and to base future decisions on these findings? 
We certainly hope this Is the case. This Is your last chance to Influence the 
outcome of the evaluation, so make sure that you use this opportunity effectively. 

Worksheet D4 deals with this element. It Is likely, however, that you 
are not now In a position to complete the worksheet as you read this workbook 
for the first time. It's all right for you to wait until a more appropriate 
time time to complete Worksheet D4. Use the worksheet to jot down some prelimi- 
nary thoughts on this issue. We've also provided an additional Appendix work- 
sheet labeled M D4A, M so that you can go back and do a more thorough job later 
on. We've gone ahead, however, and completed the sample worksheet (Figure 28). 

(Insert Figure 28) 
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Epilogue 

You may be someone who Is not currently participating In an evaluation. 
Perhaps you have read this workbook simply to Increase your understanding 
of evaluation. If this Is the case, you may also benefit from the preceding 
discussion. We urge you to reflect on the factor clusters and consider how 
they might apply In situations you have encountered In the past. Consider 
also what you might do In the future to make yourself more effective as a 
project administrator In organizing for evaluation use. 

Or perhaps you are not an administrator at all. Perhaps you are an 
evaluator, reading this book to see what kinds of actions administrators 
may take In organizing your evaluation. We hope that the material presented 
In this book has forced you to take a fresh look at the whole evaluation 
enterprise, that It has given you some new Ideas about how to perform 
evaluations that will make a difference. You will find that It Is very 
gratifying to know that your evaluation efforts will have some payoff, that 
they will amount to more than a report gathering dust on a bookshelf. 

To those researchers or evaluation theorists who picked up this book 
out of curiosity, we can say that we hope you have not been put off by the 
absence of technical language and of the scholarly apparatus of footnotes. 
He hope that you have found some Ideas here that will Inspire and Inform 
your own future work. 

Finally, to those of y6u who are our primary audience, administrators 
who are presently or will soo.i be Involved In organizing an evaluation, 

we want to encourage you in applying your own Increased capacity for. presiding over 
a meaningful, useful evaluation. He hope that your new skills and Insights 
Into the processes ©/devaluation will continue to serve you throughout 
your administrative career. 
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figure. 35 



Evaluation Topic: Jotutt JWuo 
CMWUCJuhlmj 



Factor Cluster D: 



Evaluation reporting 



Evaluator characteristics (selected) 
Information dialogue— summative 
User commitment to use 



With respect to my own program and the above topic, I ne ed to keep the following in mind: 
Factor Affecting Ust: Relevant information: Thlnge I Can Do To Influence Uae: 



D»t Evaluation 
reporting 

(frequency; timing; format 
of report; statistical/nar- 
rative data; format of 
presentation) 



3. 



2. Jht"Wut<4houliAiuwc.<nc 
^rtfeoi** jutott Miiwcui owUu 
J**ai jun*w»o /utofc unit 

3. dKiuJuwljLkouinMAt . 



2. ^cJaiuu JKctvaiuaM'unjuUu^ 



4. 



129 



Organizing For Evaluation Use 



Evaluation Topic: JUaalMtJjso 



Factor Cluster D: 



Evaluation reporting 



Evaluator characteristics (selected) 

Information dialogue— summative 
User commitment to use 



With respect to my own program and the above topic, I need to keep the following in mind: 



Factor Affecting Use: Relevant Information: 



D-2. Evaluator char- 
acteristics (selected) 

(commitment to use; 
political sensitivity) 



1. a^JwolwuVbjift eawwutttol Jo 
but m> jmhumtJo our» 



2. 



5. 
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Things I Can Do To Influence Use: 



1- \ak,MHL lyjUoMofoJotihctLL 
Motibu* Jo Jtt txMkufru 



2. 



3. 
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Evaluation Topic: JocoA J/uitio 



Factor Cluster D: 



Evaluation reporting 



Evaluator characteristics (selected) 
Information dialogue— summative 
User commitment to use 



**** 



With respect to my own program and the above topic, I need to keep the following in mind: 



Factor Affecting Um: Relevant Information: 



Things I Can Do Tc Influence U»t: 



D-3. Information 
dialogue— summative 

(amount, type and quality 
of interaction between 
evaluator and user(s)) 



^oohL Jo Jk Jwat vuiit -pu* jmfoud o*4 Athonu^okutLo^ 
out out ittin^l .W- cAtffe. ]fljiWik,ik. 



2. OlljmtJb^kjn)*^ 



2. CKac^ftft JKa euiiMflk>M 



3. 
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Wosfeteet D4 



Evaluation Topic: }om£ Mudao 
vunruculutu 



Factor Cluster D: 



Evaluation reporting 



Evatuator characteristics (selected) 
Information dialogue— summative 
User commitment to us* 



With respect to my own program and the above topic, I need to keep the following in mind: 



Factor Affecting Use: Relevant Information: 



Things I Can Do To Influence Use: 



D-4. User commitment 1. AcUc&ou J\oo Jo it mU 1. )&a»t<L MUivht, MbwmJhwqkt 

to use Utofcy tvducJlri\ 9 JMmdAd£dw& 

oo v|y>wM£fet Jhj Jul eualuflVt 
3. 3. %fet jual ^ iuuU a 

evaiuahauluwUriao 
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VUbatehaet C1A 



Evaluation Topic: 



Factor Cluster C: 



Evaluation procedures—execution 



Substance of evaluation information 



Evaluate* commitment to use 



Information dialogue—formative 



User information processing preferences 



With respect to my own program and the above topic, I need to keep the following in mind: 



Factor Affecting Use: 



C-t Evaluation 
procedures— execution 
(methods to be used; 
dealing with mandated 
tasks) 



Relevant Information: 



1. 



Things I Can Do To Influence Use: 



1. 



2. 



3. 
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Worksheet C2A 



Evaluation Topic: 



Factor Cluster C: 



Evaluation procedures—execution 



Substance of evaluation information 



Evaluate* commitment to use 



Information dialogue— formative 



User information processing preferences 



With respect to my own program and the above topic, I need to keep the following in mind: 



Factor Affecting Use: Relevant Information: 



C-2. Substance of 
evaluation Information 
(specific information 
selected; information 
relevance to audience) 



1. 



Things I Can Do To Influence Use: 



1. 



2. 



3. 
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Evaluation Topic: 



Factor Cluster C: 



Evaluation procedures— execution 



Substance of evaluation information 



Evaluates commitment to uae 



Information dialogue— formative 



User information processing preferences 



With respect to my own program and the above topic, I need to keep the following in mind: 



Factor After*! Uae: 



C-3. Evaluate* 
oommninefn 10 uae 



Re levant Information: 



1. 



Thing* I Can Do Tb Influence Uae: 



1. 



2. 



3. 



3. 



5. 
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W© steteeS C4A 



Evaluation Topic: 



Factor Cluster C: 



Evaluation procedures— execution 



Substance of evaluation information 



Evaluator commitment to use 



Information dialogue— formative 



User information processing preferences 



With respect to my own program and the above topic, I need to keep the following in mind:- 



Factor Affecting Use: 



C-4. Information 
dialogue— formative 

(amount & quality of inter- 
action between evaluator 
and user(s)) 



Relevant Information: 



Things I Can Do To Influence Use: 



3. 



4. 



5. 
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Wofifeitesrl C5A 



Evaluation Topic: 



Factor Cluster C: 



Evaluation procedures — execution 



Substance of evaluation information 



Evaluator commitment to use 



Information dialogue — formative 



User information processing preferences 



With respect to my own program and the above topic, I need to keep the following in mind: 



Factor Affecting Use: Relevant Information: 



C-5. User information 1 . 
processing preferences 



2. 



Things I Can Do To Influence Use: 



1. 



3. 



3. 



4. 



4. 



5. 



5. 
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WoteteeS DM 



Evaluation Topic: 



Factor Cluster D: 



Evaluation reporting 



Evaluator characteristics (selected) 
Information dialogue— -summative 
User commitment to use 



With respect to my own program and the above topic, I need to keep the following in mind: 



Factor Affecting Use: Relevant Information: 



0*1. Evaluation 
reporting 

(frequency; timing; format 
of report; statistical/nar- 
rative data; format of 
presentation) 



1. 



2. 



3. 



5. 



Things I Can Do To Influence Uss: 



1. 



5. 
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\Rtatesh8@t D2A 



Evaluation Topic: 



Factor Cluster D: 



Evaluation reporting 



Evaluator characteristics (selected) 

Information dialogue— summative 
User commitment to use 



With respect to my own program and the above topic, I need to keep the following in mind: 



Factor Affecting Use: Relevant Information: 



D-2. Evaluator char- 
acteristics (selected) 

(commitment to use; 
political sensitivity) 



1. 



Things I Can Do To Influence Use: 



1. 



3. 



4. 
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Worksheet D3A 



Evaluation Topic: 



Factor Cluster D: 



Evaluation reporting 



Evaluator characteristics (selected) 
Information dialogue— summatlve 
User commitment to use 



******** **** 



With respect to my own program and the above topic, I need to keep the following in mind: 
Factor Affecting Use: Relevant Information: Things I Can Do To Influence Use: 



D-3. Information 
dialogue— summatlve 

(amount, type and quality 
of interaction between 
evaluator and user(s)) 



1. 



4. 



5. 
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Woirkste'33: 




Evaluation Topic: 



Factor Cluster D: 



Evaluation reporting 



Evaiuator characteristics (selected) 
Information dialogue— summative 
User commitment to use 



With respect to my own program and the above topic, I need to keep the following in mind: 



Factor Affecting Use: Relevant Information: 



D-4. User commitment 1 . 
to use 



2. 



Things I Can Do To Influence Use: 



2. 



5. 
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